Alleviating congestion in a virtual network deployed over public clouds for an entity

ABSTRACT

Some embodiments provide a novel method for deploying different virtual networks over several public cloud datacenters for different entities. For each entity, the method (1) identifies a set of public cloud datacenters of one or more public cloud providers to connect a set of machines of the entity, (2) deploys managed forwarding nodes (MFNs) for the entity in the identified set of public cloud datacenters, and then (3) configures the MFNs to implement a virtual network that connects the entity&#39;s set of machines across its identified set of public cloud datacenters. In some embodiments, the method identifies the set of public cloud datacenters for an entity by receiving input from the entity&#39;s network administrator. In some embodiments, this input specifies the public cloud providers to use and/or the public cloud regions in which the virtual network should be defined. Conjunctively, or alternatively, this input in some embodiments specifies actual public cloud datacenters to use.

BACKGROUND

Today, a corporate enterprise network is the communication backbone thatsecurely connects the different offices and divisions of a corporation.This network is typically a wide area network (WAN) that connects (1)users in branch offices and regional campuses, (2) corporate datacentersthat host business applications, Intranets and their corresponding data,and (3) the global Internet through corporate firewalls and DMZ(demilitarized zone). Enterprise networks include specialized hardwaresuch as switches, routers and middlebox appliances interconnected byexpensive leased lines, such as Frame Relay and MPLS (multiprotocollabel switching).

In the last several years, there has been a paradigm shift in the waycorporations serve and consume communication services. First, themobility revolution has allowed users to access services from any placeat any time using mobile devices, mostly smart phones. Such users accessthe business services through public Internet and cellular networks. Atthe same time, third-party SaaS (Software as a Service) vendors (e.g.,Salesforce, Workday, Zendesk) have replaced traditional on-premiseapplications, while other applications hosted in private datacentershave been relocated to the public clouds. While this traffic is stillcarried within the enterprise network, a significant portion of itoriginates and terminates outside the corporate network perimeters andhas to cross both the public Internet (once or twice) as well as thecorporate network. Recent studies have shown that 40% of corporatenetworks report that the percentage of backhauled traffic (i.e., ofInternet traffic observed in the corporate network) is above 80%. Thismeans that the majority of the corporate traffic is carried over bothexpensive leased lines and the consumer Internet.

As a consumer-centric service, the Internet itself is a poor medium forbusiness traffic. It lacks the reliability, QoS (quality of service)guarantees and security expected by critical business applications.Moreover, the ever-increasing consumer traffic demands, net-neutralityregulations and the creation of Internet bypasses by major players(e.g., Netflix, Google, public clouds) have lowered the monetary returnper traffic unit. These trends have reduced the incentives of serviceproviders to quickly catch up with the consumer demands and offeradequate business services.

Given the growth of public clouds, corporations are migrating more oftheir compute infrastructure to the public cloud datacenters. Publiccloud providers have been at the forefront of compute and networkinginfrastructure investment. These cloud services have built manydatacenters across the world, with Azure, AWS, IBM and Google expandingto 38, 16, 25, and 14 worldwide regions respectively in 2016. Eachpublic cloud provider has interconnected its own datacenters by usingexpensive high-speed networks that employ dark fiber and undersea cablesdeployed by submarines.

Today, notwithstanding these changes, corporate network policies oftenforce all corporate traffic to go through their secure WAN gateways. Asusers become mobile and applications migrate to SaaS and public clouds,corporate WANs become costly detours that slow down all corporatecommunications. Most corporate WAN's traffic is either sourced from ordestined to the Internet. Alternate secure solutions that send thistraffic through the Internet are not adequate because of their poor andunreliable performance.

BRIEF SUMMARY

Some embodiments establish for an entity a virtual network over severalpublic cloud datacenters of one or more public cloud providers in one ormore regions (e.g., several cities, states, countries, etc.). Examplesof entities for which such a virtual network can be established includea business entity (e.g., a corporation), a non-profit entity (e.g., ahospital, a research organization, etc.), and an educational entity(e.g., a university, a college, etc.), or any other type of entity.Examples of public cloud providers include Amazon Web Services (AWS),Google Cloud Platform (GCP), Microsoft Azure, etc.

In some embodiments, high-speed, reliable private networks interconnecttwo or more of the public cloud datacenters (the public clouds). Someembodiments define the virtual network as an overlay network that spansacross several public clouds to interconnect one or more privatenetworks (e.g., networks within branches, divisions, departments of theentity or their associated datacenters), mobile users, SaaS (Software asa Service) provider machines, machines and/or services in the publiccloud(s), and other web applications.

Some embodiments utilize a logically centralized controller cluster(e.g., a set of one or more controller servers) that configures thepublic-cloud components to implement the virtual network over severalpublic clouds. In some embodiments, the controllers in this cluster areat various different locations (e.g., are in different public clouddatacenters) in order to improve redundancy and high availability. Thecontroller cluster in some embodiments scales up or down the number ofpublic cloud components that are used to establish the virtual network,or the compute or network resources allocated to these components.

Some embodiments establish different virtual networks for differententities over the same set of public clouds of the same public cloudproviders and/or over different sets of public clouds of the same ordifferent public cloud providers. In some embodiments, a virtual networkprovider provides software and services that allow different tenants todefine different virtual networks over the same or different publicclouds. In some embodiments, the same controller cluster or differentcontroller clusters can be used to configure the public cloud componentsto implement different virtual networks over the same or different setsof public clouds for several different entities.

To deploy a virtual network for a tenant over one or more public clouds,the controller cluster (1) identifies possible ingress and egressrouters for entering and exiting the virtual network for the tenantbased on locations of the tenant's branch offices, datacenters, mobileusers, and SaaS providers, and (2) identifies routes that traverse fromthe identified ingress routers to the identified egress routers throughother intermediate public-cloud routers that implement the virtualnetwork. After identifying these routes, the controller clusterpropagates these routes to the forwarding tables of the virtual networkrouters in the public cloud(s). In the embodiments that use Open vSwitch(OVS) based virtual network routers, the controller distributes theroutes by using OpenFlow.

Some embodiments provide a novel method for deploying different virtualnetworks over several public cloud datacenters for different entities.For each entity, the method (1) identifies a set of public clouddatacenters of one or more public cloud providers to connect a set ofmachines of the entity, (2) deploys managed forwarding nodes (MFNs) forthe entity in the identified set of public cloud datacenters, and then(3) configures the MFNs to implement a virtual network that connects theentity's set of machines across its identified set of public clouddatacenters.

Managed forwarding nodes in some embodiments include one or more modulesexecuting on a set of one or more host computers to perform forwardingoperations. The MFNs in some embodiments perform other operations aswell, such as service operations, e.g., NAT operations, load balancingoperations, etc. In some embodiments, the MFNs that the method deploysfor an entity are just used to process data message flows for thatentity's machines. For instance, in some embodiments, the deployed MFNsfor an entity are dedicated MFNs as they only carry the data messageflows for that entity.

In some embodiments, the different virtual networks that the methoddefines for the different entities can differ in the public clouddatacenters that they span, the public clouds of different public cloudproviders that they use and/or different public cloud regions in whichthey are defined. In some embodiments, the entity's machines that areconnected by its virtual network are machines outside of any publiccloud. In other embodiments, some of the entity's machines are in apublic cloud, while other machines reside outside of the public clouds.Also, in some embodiments, the entity's machines include SaaS providermachines that the entity uses for certain SaaS operations.

In some embodiments, the method identifies the set of public clouddatacenters for an entity by receiving input from the entity's networkadministrator. In some embodiments, this input specifies the publiccloud providers to use and/or the public cloud regions in which thevirtual network should be defined. Conjunctively, or alternatively, thisinput in some embodiments specifies the actual public cloud datacentersto use. Under the above-described approach, different entities often endup with very different virtual networks as the entities often providedifferent input regarding the desired public cloud providers, regionsand/or datacenters. The method of some embodiments supplements the setof public cloud datacenters identified for an entity through theentity's input with one or more public cloud datacenters that the methodidentifies as desirable datacenters to add to the entity's set ofdatacenters, as further described below.

To configure the MFNs, the method deploys measurement agents in publiccloud datacenters (PCDs), and has these agents exchange messages inorder to generate network measurements that quantify the quality ofnetwork connections between different pairs of PCDs or different pairsof PCD groups (e.g., between different public cloud regions oravailability zones). Examples of network measurements that the methodgenerates for a connection between two PCDs or PCD groups include loss,delay, and jitter experienced on this connection, as well as thereliability and cost of the connection.

In some embodiments, the method configures the MFNs deployed for anentity based on the network measurements that it generates for the setof PCDs that it identifies for the entity. For instance, in someembodiments, the method configures an entity's deployed MFNs by (1)performing path-identifying processes that use the measurementsgenerated for the entity's identified set of PCDs to identify a set ofpaths connecting the entity's machines across the identified set ofPCDs, and (2) using the identified paths to define next hop records thatconfigure the MFNs to forward data message flows along the differentpaths. To configure the MFNs, the method in some embodiments providesthe next hop records to a set of controllers that distribute them to theMFNs.

To identify the paths, the method uses the generated measurements toperform smallest cost (e.g., shortest) path searches. Some embodimentsallow different entities to direct the method to use different types ofmeasurements in performing its path searches, e.g., one entity candirect the method to minimize message delay, another entity can directthe method to minimize message jitter, still another entity can directthe method to minimize loss and delay, etc. For each entity, the methodin some embodiments custom configures its path search operations tooptimize a set of criteria specified by the entity.

As mentioned above, the method of some embodiments supplements the setof public cloud datacenters that it identifies for an entity through theentity's input, with one or more PCDs or PCD groups that the methodidentifies as desirable datacenters to add to the entity's set ofdatacenters. In some embodiments, the method provides to an entity arecommendation to add certain public clouds, PCD groups or PCDs to thelist of public clouds or PCDs specified by entity, before deploying anyMFN for the entity. Conjunctively, or alternatively, the method in someembodiments provides a recommendation to add one or more public cloudsor PCDs after deploying the entity's MFNs, collecting statisticsregarding their usage and analyzing the statistics to determine that itis desirable to add the public clouds or PCDs. Based on its analysis ofthe collected statistics, the method in some embodiments also recommendsremoval of underutilized public clouds or PCDs.

The method of some embodiments allows an entity to use other MFNs thathave not been specifically deployed for the entity under certaincircumstances. These other MFNs in some embodiments are shared MFNs asmultiple entities can use them, as opposed to dedicated MFNs that arespecifically deployed for a single entity. In some embodiments, thededicated MFNs that are specifically deployed for one entity have thesame attributes and perform the same operations (e.g., check for atenant identifier in performing its forwarding operations) as the sharedMFNs that are deployed and used by multiple entities. The onlydifference between the dedicated and shared MFNs in these embodiments isthat the dedicated MFNs are used to process data messages for just onetenant, while the shared MFNs are used to process data messages formultiple entities.

One example in which an entity can use a shared MFN that has not beenspecifically deployed for the entity involves the use of MFNs in remotelocations by the mobile devices of an entity in those location. Forinstance, an entity may predominantly operate in one region (e.g., onlyhave offices in North America, etc.), but may have users that go ontrips internationally and need to access the entity's network throughthe virtual network that is deployed for it over the public clouds. Forsuch situations, the method of some embodiments allows the mobiledevices (e.g., phones, tablets, laptops, etc.) of these traveling usersto access its virtual network through an MFN that the method deploys inone or more PCDs (e.g., public clouds in Europe or Asia) in the foreigncountries.

Another example involves the use of MFNs that are deployed indatacenters near the Internet backbone or serving as part of theInternet backbone. Setting up machines near or at the Internet backboneis difficult and can be expensive for any one entity. Accordingly,entities might not typically request that their virtual networks have adedicated MFN deployed near or at the Internet backbone. Deploying suchdedicated MFNs might not even be possible.

Before or after deploying the MFNs for an entity, the method's pathsearches might determine that for certain compute nodes of the entity,it is desirable to use a path that traverses through one or more sharedMFNs deployed near or at the Internet backbone (i.e., to use a path thatleaves the dedicated virtual network of the entity to use one or moreMFNs near or at the Internet backbone). In such cases, the method ofsome embodiments provides a recommendation to the entity that the sharedMFNs deployed near or at the Internet backbone should be used. When theentity accepts this recommendation, the method configures the MFNs touse the identified path(s) that use the shared MFNs near or at theInternet backbone.

The method of some embodiments allows for temporary usage of the sharedMFNs when an entity's dedicated virtual network appears congested or isexpected to be congested at one or more MFNs that are specificallydeployed for the entity. In some embodiments, the method collects andanalyzes statistics regarding the use of the dedicated MFNs that havebeen deployed for a particular entity. Based on this analysis, themethod identifies one or more MFNs that are congested, and in response,reconfigures one or more network elements (e.g., load balancers) toredirect some of the data message flows to the shared MFNs andreconfigures these shared MFNs to forward the entity's data messageflows until they reach their destination nodes or they reach anotheringress node into the entity's virtual network.

The method in some embodiments does not deploy and configure MFNs toimplement virtual networks. For instance, in some embodiments, themethod provides measurements that quantify connections between PCDs orPCD groups to other processes that deploy and configure MFNs. In otherembodiments, the method provides these measurements to other processesthat perform other cloud-based operations, such as processes that deployapplication machines in the public clouds and use the measurements toidentify the best locations for such deployments. The method of stillother embodiments uses these measurements to perform other cloud-basedoperations itself (e.g., deploying application machines in the publicclouds and using the measurements to identify the best locations forsuch deployments).

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description, the Drawings and the Claims isneeded. Moreover, the claimed subject matters are not to be limited bythe illustrative details in the Summary, Detailed Description and theDrawing.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purposes of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1A presents a virtual network that is defined for a corporationover several public cloud datacenters of two public cloud providers.

FIG. 1B illustrates an example of two virtual networks for two corporatetenants that are deployed over the public clouds.

FIG. 1C alternatively illustrates an example of two virtual networks,with one network deployed over public clouds and the other virtualnetwork deployed over another pair of public clouds.

FIG. 2 illustrates an example of a managed forwarding node and acontroller cluster of some embodiments of the invention.

FIG. 3 illustrates an example of a measurement graph that the controllermeasurement-processing layer produces in some embodiments.

FIG. 4A illustrates an example of a routing graph that the controllerpath-identifying layer produces in some embodiments from the measurementgraph.

FIG. 4B illustrates an example of adding known IPs for two SaaSproviders to the two nodes in the routing graph that are in datacentersthat are closest to the datacenters of these SaaS providers.

FIG. 4C illustrates a routing graph that is generated by adding twonodes to represent two SaaS providers.

FIG. 4D illustrates a routing graph with additional nodes added torepresent branch offices and datacenters with known IP addresses thatconnect respectively to two public clouds.

FIG. 5 illustrates a process that the controller path-identifying layeruses to generate a routing graph from a measurement graph received fromthe controller measurement layer.

FIG. 6 illustrates the IPsec data message format of some embodiments.

FIG. 7 illustrates an example of the two encapsulating headers of someembodiments, while FIG. 8 presents an example that illustrates how thesetwo headers are used in some embodiments.

FIGS. 9-11 illustrate message-handling processes that are performedrespectively by the ingress, intermediate, and egress managed forwardingnodes (MFNs) when they receive a message that is sent between twocompute devices in two different branch offices.

FIG. 12 illustrates an example that does not involve an intermediate MFNbetween the ingress and egress MFNs.

FIG. 13 illustrates a message-handling process that is performed by thecloud forwarding element (CFE) of the ingress MFN when it receives amessage that is sent from a corporate compute device in a branch officeto another device in another branch office or in a SaaS providerdatacenter.

FIG. 14 presents an example that shows M virtual corporate WANs for Mtenants of a virtual network provider that has network infrastructureand controller cluster(s) in N public clouds of one or more public cloudproviders.

FIG. 15 conceptually illustrates a process performed by the controllercluster of the virtual network provider to deploy and manage a virtualWAN for a particular tenant.

FIG. 16 illustrates a three-layer SaaS deployment model of someembodiments.

FIG. 17 illustrates a two-layer SaaS deployment model of someembodiments.

FIG. 18 illustrates a process used by the central controller cluster ofsome embodiments to define routes for a multi-homed, multi-machinecompute node (MMCN).

FIG. 19 presents an example of two branch nodes of two MMCNs and a SaaSdatacenter.

FIG. 20 illustrates a process used by the central controller cluster ofsome embodiments to define routes for multi-homed SaaS providers.

FIG. 21 illustrates a process that the VNP uses in some embodiments todeploy and configure dedicated MFNs to establish a dedicated virtualnetwork for an entity that requests such a network to be deployed over aparticular set of public cloud providers, a particular set of publiccloud regions, and/or a particular set of public cloud datacenters.

FIG. 22 presents an example that illustrates three different virtualnetworks deployed over several public clouds in the United States forthree different companies.

FIG. 23 conceptually illustrates a VNP process that producesrecommendations to add MFNs based on collected statistics and newmeasurements.

FIG. 24 illustrates an example of adding a new PCD to a virtual networkto improve its performance.

FIG. 25 conceptually illustrates a VNP process that producesrecommendations to remove one or more underutilized MFNs.

FIG. 26 illustrates an example of removing a PCD from a virtual networkin order to remove an underutilized MFN in this PCD.

FIG. 27 conceptually illustrates a VNP process that producesrecommendations to offload some of the data message traffic fromdedicated MFNs of a dedicated virtual network of an entity to one ormore shared MFNs for at least a duration of at least one path between atleast one pair of machine endpoints.

FIGS. 28-30 illustrate example of directing some of the data messageflows away from over congested dedicated MFNs of a dedicated virtualnetwork to shared MFNs.

FIG. 31 illustrates a process that stops the redirection of some of thedata traffic load away from a dedicated MFN.

FIG. 32 conceptually illustrates a computer system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments establish for an entity a virtual network over severalpublic cloud datacenters of one or more public cloud providers in one ormore regions (e.g., several cities, states, countries, etc.). An exampleof an entity for which such a virtual network can be established includea business entity (e.g., a corporation), a non-profit entity (e.g., ahospital, a research organization, etc.), and an educational entity(e.g., a university, a college, etc.), or any other type of entity.Examples of public cloud providers include Amazon Web Services (AWS),Google Cloud Platform (GCP), Microsoft Azure, etc.

Some embodiments define the virtual network as an overlay network thatspans across several public cloud datacenters (public clouds) tointerconnect one or more private networks (e.g., networks withinbranches, divisions, departments of the entity or their associateddatacenters), mobile users, SaaS (Software as a Service) providermachines, machines and/or services in the public cloud(s), and other webapplications. In some embodiments, high-speed, reliable private networksinterconnect two or more of the public cloud datacenters.

The virtual network in some embodiments can be configured to optimizethe routing of the entity's data messages to their destinations for bestend-to-end performance, reliability and security, while trying tominimize the routing of this traffic through the Internet. Also, thevirtual network in some embodiments can be configured to optimize thelayer 4 processing of the data message flows passing through thenetwork. For instance, in some embodiments, the virtual networkoptimizes the end-to-end rate of TCP (Transport Control Protocol)connections by splitting the rate control mechanisms across theconnection path.

Some embodiments establish the virtual network by configuring severalcomponents that are deployed in several public clouds. These componentsinclude in some embodiments software-based measurement agents, softwareforwarding elements (e.g., software routers, switches, gateways, etc.),layer-4 connection proxies and middlebox service machines (e.g.,appliances, VMs, containers, etc.).

Some embodiments utilize a logically centralized controller cluster(e.g., a set of one or more controller servers) that configures thepublic-cloud components to implement the virtual network over severalpublic clouds. In some embodiments, the controllers in this cluster areat various different locations (e.g., are in different public clouddatacenters) in order to improve redundancy and high availability. Whendifferent controllers in the controller cluster are located in differentpublic cloud datacenters, the controllers in some embodiments sharetheir state (e.g., the configuration data that they generate to identifytenants, routes through the virtual networks, etc.). The controllercluster in some embodiments scales up or down the number of public cloudcomponents that are used to establish the virtual network, or thecompute or network resources allocated to these components.

Some embodiments establish different virtual networks for differententities over the same set of public clouds of the same public cloudproviders and/or over different sets of public clouds of the same ordifferent public cloud providers. In some embodiments, a virtual networkprovider provides software and services that allow different tenants todefine different virtual networks over the same or different publicclouds. In some embodiments, the same controller cluster or differentcontroller clusters can be used to configure the public cloud componentsto implement different virtual networks over the same or different setsof public clouds for several different entities.

Several examples of corporate virtual networks are provided in thediscussion below. However, one of ordinary skill will realize that someembodiments define virtual networks for other types of entities, such asother business entities, non-profit organizations, educational entities,etc. Also, as used in this document, data messages refer to a collectionof bits in a particular format sent across a network. One of ordinaryskill in the art will recognize that the term data message is used inthis document to refer to various formatted collections of bits that aresent across a network. The formatting of these bits can be specified bystandardized protocols or non-standardized protocols. Examples of datamessages following standardized protocols include Ethernet frames, IPpackets, TCP segments, UDP datagrams, etc. Also, as used in thisdocument, references to L2, L3, L4, and L7 layers (or layer 2, layer 3,layer 4, and layer 7) are references respectively to the second datalink layer, the third network layer, the fourth transport layer, and theseventh application layer of the OSI (Open System Interconnection) layermodel.

FIG. 1A presents a virtual network 100 that is defined for a corporationover several public cloud datacenters 105 and 110 of two public cloudproviders A and B. As shown, the virtual network 100 is a secure overlaynetwork that is established by deploying different managed forwardingnodes 150 in different public clouds and connecting the managedforwarding nodes (MFNs) to each other through overlay tunnels 152. Insome embodiments, an MFN is a conceptual grouping of several differentcomponents in a public cloud datacenter that with other MFNs (with othergroups of components) in other public cloud datacenters establish one ormore overlay virtual networks for one or more entities.

As further described below, the group of components that form an MFNinclude in some embodiments (1) one or more VPN gateways forestablishing VPN connections with an entity's compute nodes (e.g.,offices, private datacenters, remote users, etc.) that are externalmachine locations outside of the public cloud datacenters, (2) one ormore forwarding elements for forwarding encapsulated data messagesbetween each other in order to define an overlay virtual network overthe shared public cloud network fabric, (3) one or more service machinesfor performing middlebox service operations as well as L4-L7optimizations, and (4) one or more measurement agents for obtainingmeasurements regarding the network connection quality between the publiccloud datacenters in order to identify desired paths through the publiccloud datacenters. In some embodiments, different MFNs can havedifferent arrangements and different numbers of such components, and oneMFN can have different numbers of such components for redundancy andscalability reasons.

Also, in some embodiments, each MFN's group of components execute ondifferent computers in the MFN's public cloud datacenter. In someembodiments, several or all of an MFN's components can execute on onecomputer of a public cloud datacenter. The components of an MFN in someembodiments execute on host computers that also execute other machinesof other tenants. These other machines can be other machines of otherMFNs of other tenants, or they can be unrelated machines of othertenants (e.g., compute VMs or containers).

The virtual network 100 in some embodiments is deployed by a virtualnetwork provider (VNP) that deploys different virtual networks over thesame or different public cloud datacenters for different entities (e.g.,different corporate customers/tenants of the virtual network provider).The virtual network provider in some embodiments is the entity thatdeploys the MFNs and provides the controller cluster for configuring andmanaging these MFNs.

The virtual network 100 connects the corporate compute endpoints (suchas datacenters, branch offices and mobile users) to each other and toexternal services (e.g., public web services, or SaaS services such asOffice365 or Salesforce) that reside in the public cloud or reside inprivate datacenter accessible through the Internet. As further describedbelow, SaaS in some embodiments is a software distribution model inwhich a third-party provider hosts applications and makes them availableto customers over the Internet.

The virtual network 100 leverages the different locations of thedifferent public clouds to connect different corporate compute endpoints(e.g., different private networks and/or different mobile users of thecorporation) to the public clouds in their vicinity. Corporate computeendpoints are also referred to as corporate compute nodes in thediscussion below. In some embodiments, the virtual network 100 alsoleverages the high-speed networks that interconnect these public cloudsto forward data messages through the public clouds to their destinationsor to get as close to their destinations while reducing their traversalthrough the Internet. When the corporate compute endpoints are outsideof public cloud datacenters over which the virtual network spans, theseendpoints are referred to as external machine locations. This is thecase for corporate branch offices, private datacenters and devices ofremote users.

In the example illustrated in FIG. 1A, the virtual network 100 spans sixdatacenters 105 a-105 f of the public cloud provider A and fourdatacenters 110 a-110 d of the public cloud provider B. In spanningthese public clouds, this virtual network connects several branchoffices, corporate datacenters, SaaS providers and mobile users of thecorporate tenant that are located in different geographic regions.Specifically, the virtual network 100 connects two branch offices 130 aand 130 b in two different cities (e.g., San Francisco, Calif., andPune, India), a corporate datacenter 134 in another city (e.g., Seattle,Wash.), two SaaS provider datacenters 136 a and 136 b in another twocities (Redmond, Wash., and Paris, France), and mobile users 140 atvarious locations in the world. As such, this virtual network can beviewed as a virtual corporate WAN.

In some embodiments, the branch offices 130 a and 130 b have their ownprivate networks (e.g., local area networks) that connect computers atthe branch locations and branch private datacenters that are outside ofpublic clouds. Similarly, the corporate datacenter 134 in someembodiments has its own private network and resides outside of anypublic cloud datacenter. In other embodiments, however, the corporatedatacenter 134 or the datacenter of the branch 130 a and 130 b can bewithin a public cloud, but the virtual network does not span this publiccloud, as the corporate or branch datacenter connects to the edge of thevirtual network 100.

As mentioned above, the virtual network 100 is established by connectingdifferent deployed managed forwarding nodes 150 in different publicclouds through overlay tunnels 152. Each managed forwarding node 150includes several configurable components. As further described above andfurther described below, the MFN components include in some embodimentssoftware-based measurement agents, software forwarding elements (e.g.,software routers, switches, gateways, etc.), layer 4 proxies (e.g., TCPproxies) and middlebox service machines (e.g., VMs, containers, etc.).One or more of these components in some embodiments use standardized orcommonly available solutions, such as Open vSwitch, OpenVPN, strongSwan,etc.

In some embodiments, each MFN (i.e., the group of components theconceptually forms an MFN) can be shared by different tenants of thevirtual network provider that deploys and configures the MFNs in thepublic cloud datacenters. Conjunctively, or alternatively, the virtualnetwork provider in some embodiments can deploy a unique set of MFNs inone or more public cloud datacenters for a particular tenant. Forinstance, a particular tenant might not wish to share MFN resources withanother tenant for security reasons or quality of service reasons. Forsuch a tenant, the virtual network provider can deploy its own set ofMFNs across several public cloud datacenters.

In some embodiments, a logically centralized controller cluster 160(e.g., a set of one or more controller servers) operate inside oroutside of one or more of the public clouds 105 and 110, and configurethe public-cloud components of the managed forwarding nodes 150 toimplement the virtual network over the public clouds 105 and 110. Insome embodiments, the controllers in this cluster are at variousdifferent locations (e.g., are in different public cloud datacenters) inorder to improve redundancy and high availability. The controllercluster in some embodiments scales up or down the number of public cloudcomponents that are used to establish the virtual network, or thecompute or network resources allocated to these components.

In some embodiments, the controller cluster 160, or another controllercluster of the virtual network provider, establishes a different virtualnetwork for another corporate tenant over the same public clouds 105 and110, and/or over different public clouds of different public cloudproviders. In addition to the controller cluster(s), the virtual networkprovider in other embodiments deploys forwarding elements and servicemachines in the public clouds that allow different tenants to deploydifferent virtual networks over the same or different public clouds.FIG. 1B illustrates an example of two virtual networks 100 and 180 fortwo corporate tenants that are deployed over the public clouds 105 and110. FIG. 1C alternatively illustrates an example of two virtualnetworks 100 and 182, with one network 100 deployed over public clouds105 and 110 and the other virtual network 182 deployed over another pairof public clouds 110 and 115.

Through the configured components of the MFNs, the virtual network 100of FIG. 1A allows different private networks and/or different mobileusers of the corporate tenant to connect to different public clouds thatare in optimal locations (e.g., as measured in terms of physicaldistance, in terms of connection speed, loss, delay and/or cost, and/orin terms of network connection reliability, etc.) with respect to theseprivate networks and/or mobile users. These components also allow thevirtual network 100 in some embodiments to use the high-speed networksthat interconnect the public clouds to forward data messages through thepublic clouds to their destinations while reducing their traversalthrough the Internet.

In some embodiments, the MFN components are also configured to run novelprocesses at the network, transport and application layers to optimizethe end-to-end performance, reliability and security. In someembodiments, one or more of these processes implement proprietaryhigh-performance networking protocols, free from the current networkprotocol ossification. As such, the virtual network 100 in someembodiments is not confined by Internet autonomous systems, routingprotocols, or even end-to-end transport mechanisms.

For example, in some embodiments, the components of the MFNs 150 (1)create optimized, multi-path and adaptive centralized routing, (2)provide strong QoS (Quality of Service) guarantees, (3) optimizeend-to-end TCP rates through intermediate TCP splitting and/ortermination, and (4) relocate scalable application-level middleboxservices (e.g., firewalls, intrusion detection systems (IDS), intrusionprevention system (IPS), WAN optimization, etc.) to the compute part ofthe cloud in a global network function virtualization (NFV).Accordingly, the virtual network can be optimized to fit customized andchanging demands of the corporation without being bound to existingnetwork protocol. Also, in some embodiments, the virtual network can beconfigured as a “pay as you go” infrastructure that can be dynamicallyand elastically scaled up and down both in performance capability and ingeographical span according to the continuous requirement changes.

To implement the virtual network 100, at least one managed forwardingnode 150 in each public cloud datacenter 105 a-105 f and 110 a-110 dspanned by the virtual network has to be configured by the set ofcontrollers. FIG. 2 illustrates an example of a managed forwarding node150 and a controller cluster 160 of some embodiments of the invention.In some embodiments, each managed forwarding node 150 is a machine(e.g., a VM or container) that executes on a host computer in a publiccloud datacenter. In other embodiments, each managed forwarding node 150is implemented by multiple machines (e.g., multiple VMs or containers)that execute on the same host computer in one public cloud datacenter.In still other embodiments, two or more components of one MFN can beimplemented by two or more machines executing on two or more hostcomputers in one or more public cloud datacenters.

As shown, the managed forwarding node 150 includes a measurement agent205, firewall and NAT middlebox service engines 210 and 215, one or moreoptimization engines 220, edge gateways 225 and 230, and a cloudforwarding element 235 (e.g., a cloud router). In some embodiments, eachof these components 205-235 can be implemented as a cluster of two ormore components.

The controller cluster 160 in some embodiments can dynamically scale upor down each component cluster (1) to add or remove machines (e.g., VMsor containers) to implement each component's functionality and/or (2) toadd or remove compute and/or network resources to the previouslydeployed machines that implement that cluster's components. As such,each deployed MFN 150 in a public cloud datacenter can be viewed as acluster of MFNs, or it can be viewed as a node that includes multipledifferent component clusters that perform different operations of theMFN.

Also, in some embodiments, the controller cluster deploys different setsof MFNs in the public cloud datacenters for different tenants for whichthe controller cluster defines virtual networks over the public clouddatacenters. In this approach, the virtual networks of any two tenantsdo not share any MFN. However, in the embodiments described below, eachMFN can be used to implement different virtual networks for differenttenants. One of ordinary skill will realize that in other embodimentsthe controller cluster 160 can implement the virtual network of eachtenant of a first set of tenants with its own dedicated set of deployedMFNs, while implementing the virtual network of each tenant of a secondset of tenants with a shared set of deployed MFNs.

In some embodiments, the branch gateway 225 and remote device gateway230 establish secure VPN connections respectively with one or morebranch offices 130 and remote devices (e.g., mobile devices 140) thatconnect to the MFN 150, as shown in FIG. 2. One example of such VPNconnections are IPsec connections, which will be further describedbelow. However, one of ordinary skill will realize that in otherembodiments, such gateways 225 and/or 230 establish different types ofVPN connections.

An MFN 150 in some embodiments includes one or more middlebox enginesthat perform one or more middlebox service operations, such are firewalloperations, NAT operations, IPS operations, IDS operations, loadbalancing operations, WAN optimization operations, etc. By incorporatingthese middlebox operations (e.g., firewall operations, WAN optimizationoperations, etc.) in the MFNs that are deployed in the public cloud, thevirtual network 100 implements in the public cloud much of the functionsthat are traditionally performed by the corporate WAN infrastructure ata corporation's datacenter(s) and/or branch office(s).

Accordingly, for many of the middlebox services, the corporate computenodes (e.g., remote devices, branch offices and datacenters) no longerhave to access the corporate WAN infrastructure of the corporation in aprivate datacenter or branch office, as much of these services are nowdeployed in the public clouds. This approach speeds up the access of thecorporate compute nodes (e.g., remote devices, branch offices anddatacenters) to these services, and avoids costly congested-networkbottlenecks at private datacenters that would otherwise be dedicated tooffering such services.

This approach effectively distributes the WAN gateway functionality tovarious MFNs in the public cloud datacenters. For instance, in thevirtual network 100 of some embodiments, most or all of the traditionalcorporate WAN gateway security functions (e.g., firewall operations,intrusion detection operations, intrusion prevention operations, etc.)are moved to the public cloud MFNs (e.g., ingress MFNs at which datafrom compute endpoints is received into the virtual network). Thiseffectively allows the virtual network 100 to have a distributed WANgateway that is implemented at many different MFNs that implement thevirtual network 100.

In the example illustrated in FIG. 2, the MFN 150 is shown to includethe firewall engine 210, the NAT engine 215 and one or more L4-L7optimization engines. One of ordinary skill will realize that in otherembodiments, the MFN 150 includes other middlebox engines for performingother middlebox operations. In some embodiments, the firewall engine 210enforces firewall rules on (1) data message flows on their ingress pathsinto the virtual network (e.g., on data message flows that the gateways225 and 230 receives and process from branch offices 130 and mobiledevices 140) and (2) data messages flows on their egress paths out ofthe virtual network (e.g., on data message flows that are sent to SaaSprovider datacenters through the NAT engine 215 and the Internet 202).

The firewall engine 210 of the MFN 150 in some embodiments also enforcesfirewall rules when the firewall engine belongs to an MFN that is anintermediate hop between an ingress MFN at which a data message flowenters a virtual network and an egress MFN at which the data messageflow exits the virtual network. In other embodiments, the firewallengine 210 only enforces firewall rules when it is part of a datamessage flow's ingress MFN and/or egress MFN.

In some embodiments, the NAT engine 215 performs a network addresstranslation to change the source network addresses of data message flowson their egress paths out of the virtual network to third party devices(e.g., to SaaS provider machines) through the Internet 202. Such networkaddress translations ensure that third-party machines (e.g., SaaSmachines) can be properly configured to process the data message flowsthat without the address translations might specify private networkaddresses of the tenants and/or the public cloud providers. This isparticularly problematic as private network addresses of differenttenants and/or cloud providers might overlap. The address translationalso ensures that the reply messages from the third party devices (e.g.,the SaaS machines) can be properly received by the virtual network(e.g., by the MFN NAT engine from which the message exited the virtualnetwork).

The NAT engines 215 of the MFNs in some embodiments perform double-NAToperations on each data message flow that leaves the virtual network toreach a third party machine, or that enters the virtual network from athird party machine. As further described in U.S. Public PatentApplication 2019-0103990 A1 (incorporated herein by reference), one NAToperation in the two NAT operations is performed on such a data messageflow at its ingress MFN when it enters the virtual network, while theother NAT operation is performed on the data message flow at its egressMFN when it exits the virtual network.

This double NAT approach allows more tenant private networks to bemapped to the networks of the public cloud providers. This approach alsoreduces the load for distributing to the MFNs data regarding changes totenant private networks. Before the ingress or egress NAT operations,some embodiments perform a tenant mapping operation that uses the tenantidentifier to first map the tenant's source network address to anothersource network address that is then mapped to yet another source networkaddress by the NAT operation. Performing the double NAT operationreduces the data distribution load for distributing data regardingchanges to the tenant private networks.

The optimization engine 220 executes novel processes that optimize theforwarding of the entity's data messages to their destinations for bestend-to-end performance and reliability. Some of these processesimplement proprietary high-performance networking protocols, free fromthe current network protocol ossification. For example, in someembodiments, the optimization engine 220 optimizes end-to-end TCP ratesthrough intermediate TCP splitting and/or termination.

The cloud forwarding element 235 is the MFN engine that is responsiblefor forwarding a data message flow to the next hop MFN's cloudforwarding element (CFE) when the data message flow has to traverse toanother public cloud to reach its destination, or to an egress router inthe same public cloud when the data message flow can reach itsdestination through the same public cloud. In some embodiments, the CFE235 of the MFN 150 is a software router.

To forward the data messages, the CFE encapsulates the messages withtunnel headers. Different embodiments use different approaches toencapsulate the data messages with tunnel headers. Some embodimentsdescribed below use one tunnel header to identify network ingress/egressaddresses for entering and exiting the virtual network, and use anothertunnel header to identify next hop MFNs when a data message has totraverse one or more intermediate MFN to reach the egress MFN.

Specifically, in some embodiments, the CFE sends the data message withtwo tunnel headers (1) an inner header that identifies an ingress CFEand egress CFE for entering and exiting the virtual network, and (2) anouter header that identifies the next hop CFE. The inner tunnel headerin some embodiments also includes a tenant identifier (TID) in order toallow multiple different tenants of the virtual network provider to usea common set of MFN CFEs of the virtual network provider. Otherembodiments define tunnel headers differently in order to define theoverlay virtual network.

To deploy a virtual network for a tenant over one or more public clouds,the controller cluster (1) identifies possible ingress and egressrouters for entering and exiting the virtual network for the tenantbased on locations of the tenant's corporate compute nodes (e.g., branchoffices, datacenters, mobile users and SaaS providers), and (2)identifies routes that traverse from the identified ingress routers tothe identified egress routers through other intermediate public-cloudrouters that implement the virtual network. After identifying theseroutes, the controller cluster propagates these routes to the forwardingtables of the MFN CFEs 235 in the public cloud(s). In the embodimentsthat use OVS-based virtual network routers, the controller distributesthe routes by using OpenFlow.

In some embodiments, the controller cluster 160 can also configure thecomponents 205-235 of each MFN 150 that implements the virtual networkto optimize several network processing layers in order to achieve bestend-to-end performance, reliability and security. For example, in someembodiments, these components are configured (1) to optimize layer3traffic routing (e.g., shortest path, packet duplication), (2) tooptimize layer 4 TCP congestion control (e.g., segmentation, ratecontrol), (3) to implement security features (e.g., encryption, deeppacket inspection, firewall), and (4) to implement application-layercompression features (e.g., de-duplication, caching). Within the virtualnetwork, corporate traffic is secured, inspected and logged.

In some embodiments, one measurement agent is deployed for each MFN in apublic cloud datacenter. In other embodiments, multiple MFNs in a publiccloud datacenter or in a collection of datacenters (e.g., in acollection of nearby, associated datacenters, such as datacenters in oneavailability zone) share one measurement agent. To optimize the layers 3and 4 processing, the measurement agent 205 associated with each managedforwarding node 150 repeatedly generates measurement values thatquantify the quality of the network connection between its node and eachof several other “neighboring” nodes.

Different embodiments define neighboring nodes differently. For aparticular MFN in one public cloud datacenter of a particular publiccloud provider, a neighboring node in some embodiments includes (1) anyother MFN that operates in any public cloud datacenter of the particularpublic cloud provider, and (2) any other MFN that operates in anotherpublic cloud provider's datacenter that is within the same “region” asthe particular MFN.

Different embodiments define the same region differently. For instance,some embodiments define a region in terms of a distance that specifies abounding shape around the particular managed forwarding node. Otherembodiments define regions in terms of cities, states, or regionalareas, such as northern California, southern California, etc. Theassumption of this approach is that different datacenters of the samepublic cloud provider are connected with very high-speed networkconnections, while the network connections between the datacenters ofdifferent public cloud providers are likely fast when the datacentersare within the same region but likely not as fast when the datacentersare in different regions. The connection between the datacenters ofdifferent public cloud providers might have to traverse long distancesthrough the public Internet when the datacenters are in differentregions.

The measurement agent 205 generates measurement values differently indifferent embodiments. In some embodiments, the measurement agent sendspinging messages (e.g., UDP echo messages) periodically (e.g., onceevery second, every N seconds, every minute, every M minutes, etc.) toeach of the measurement agents of its neighboring managed forwardingnodes. Given the small size of the pinging messages, they do not resultin large network connection charges. For instance, for 100 nodes witheach node sending a ping to each other node every 10 seconds, about 10Kb/s of ingress and egress measurement traffic is generated for eachnode, and this leads to network consumption charges of a few dollars(e.g., $5) per node per year, given the current public cloud prices.

Based on the speed of the reply messages that it receives, themeasurement agent 205 computes and updates measurement metric values,such as network-connection throughput speed, delay, loss, and linkreliability. By repeatedly doing these operations, the measurement agent205 defines and updates a matrix of measurement results that expressesthe quality of network connections to its neighboring nodes. As theagent 205 interacts with the measurement agents of its neighboringnodes, its measurement matrix only quantifies the quality of theconnections to its local clique of nodes.

The measurement agents of the different managed forwarding nodes sendtheir measurement matrices to the controller cluster 160, which thenaggregates all different clique connection data to obtain an aggregatemesh view of the connections between different pairs of managedforwarding nodes. When the controller cluster 160 collects differentmeasurements for a link between two pairs of forwarding nodes (e.g.,measurements taken by one node at different times), the controllercluster produces a blended value from the different measurements (e.g.,produces an average or a weighted average of the measurements). Theaggregate mesh view in some embodiments is a full mesh view of all thenetwork connections between each pair of managed forwarding nodes, whilein other embodiments it is a more complete view than the one produced bythe measurement agents of the individual managed forwarding nodes.

As shown in FIG. 2, the controller cluster 160 includes a cluster of oneor more measurement-processing engines 280, one or more path-identifyingengines 282, and one or more management interfaces 284. In order not toobscure the description with unnecessary detail, each of these clusterswill be referred to below in terms of singular engine or interfacelayers, i.e., in terms of a measurement-processing layer 280, apath-identifying layer 282, and a management interface layer 284.

The measurement-processing layer 280 receives the measurement matricesfrom the measurement agents 205 of the managed forwarding nodes andprocesses these measurements matrices to produce the aggregate meshmatrix that expresses the connection quality between different pairs ofmanaged forwarding nodes. The measurement-processing layer 280 providesthe aggregate mesh matrix to the path-identifying layer 282. Based onthe aggregate mesh matrix, the path-identifying layer 282 identifiesdifferent desired routing paths through the virtual network forconnecting different corporate data endpoints (e.g., different branchoffices, corporate datacenters, SaaS provider datacenters and/or remotedevices). This layer 282 then provides these routing paths in routetables that are distributed to the cloud forwarding elements 235 of themanaged forwarding nodes 150.

In some embodiments, the identified routing path for each pair of datamessage endpoints is a routing path that is deemed optimal based on aset of optimization criteria, e.g., it is the fastest routing path, theshortest routing path, or the path that least uses the Internet. Inother embodiments, the path-identifying engine can identify and provide(in the routing table) multiple different routing paths between the sametwo endpoints. In these embodiments, the cloud forwarding elements 235of the managed forwarding nodes 150 then select one of the paths basedon QoS criteria or other runtime criteria that they are enforcing. EachCFE 235 in some embodiments does not receive the entire routing pathfrom the CFE to the egress point of the virtual network, but ratherreceives the next hop for the path.

In some embodiments, the path-identifying layer 282 uses the measurementvalues in the aggregate mesh matrix as inputs to routing algorithms thatit executes to construct a global routing graph. This global routinggraph is an aggregated and optimized version of a measurement graph thatthe measurement-processing layer 280 produces in some embodiments. FIG.3 illustrates an example of a measurement graph 300 that the controllermeasurement-processing layer 280 produces in some embodiments. Thisgraph depicts network connections between various managed forwardingnodes 150 in AWS and GCP public clouds 310 and 320 (i.e., in thedatacenters of AWS and GCP). FIG. 4A illustrates an example of a routinggraph 400 that the controller path-identifying layer 282 produces insome embodiments from the measurement graph 300.

FIG. 5 illustrates a process 500 that the controller path-identifyinglayer uses to generate a routing graph from a measurement graph receivedfrom the controller measurement layer. The path-identifying layer 282performs this process 500 repeatedly as it repeatedly receives updatedmeasurement graphs from the controller measurement layer (e.g., performsthe process 500 each time that it receives a new measurement graph, oreach N^(th) time that it receives a new measurement graph). In otherembodiments, the path-identifying layer 282 performs this processperiodically (e.g., once every 12 hours or 24 hours).

As shown, the path-identifying layer initially defines (at 505) therouting graph to be identical to the measurement graph (i.e., to havethe same links between the same pairs of managed forwarding nodes). At510, the process removes bad links from the measurement graph 300.Examples of bad links are links with excessive message loss or poorreliability (e.g., links with greater than 2% message loss in last 15minutes, or with message loss greater than 10% in the last 2 minute).FIG. 4A illustrates that links 302, 304 and 306 in the measurement graph300 are excluded in the routing graph 400. This figure illustrates theexclusion of these links by depicting these links with dashed lines.

Next, at 515, the process 500 computes a link weight score (cost score)as a weighted combination of several computed and provider-specificvalues. In some embodiments, the weight score is a weighted combinationof the link's (1) computed delay value, (2) computed loss value, (3)provider network-connection cost, and (4) provider compute cost. In someembodiments, the provider compute cost is accounted for as the managedforwarding nodes connected by the link are machines (e.g., VMs orcontainers) that execute on host computers in the public clouddatacenter(s).

At 520, the process adds to the routing graph the known source anddestination IP addresses (e.g., known IPs of SaaS providers used by thecorporate entity) for the data message flows in the virtual network. Insome embodiments, the process adds each known IP address of a possiblemessage-flow endpoint to the node (e.g., to the node representing anMFN) in the routing graph that is closest to that end point. In doingso, the process in some embodiments assumes that each such endpoint isconnected to the virtual network through a link with a zero delay costand a zero loss cost. FIG. 4B illustrates an example of adding known IPsfor two SaaS providers to the two nodes 402 and 404 (representing twoMFNs) in the routing graph that are in datacenters that are closest tothe datacenters of these SaaS providers. In this example, one node is inan AWS public cloud, while the other node is in the GCP public cloud.

Alternatively, or conjunctively, the process 500 in some embodimentsadds the known source and destination IP addresses to the routing graphby adding nodes to this graph to represent the source and destinationendpoints, assigning IP addresses to these nodes, and assigning weightvalues to the links that connect these added nodes to other nodes in therouting graph (e.g., to nodes in the routing graph that represent MFNsin the public clouds). When the source and destination endpoints for theflows are added as nodes, the path-identifying engine 282 can accountfor cost (e.g., distance cost, delay cost, and/or financial cost, etc.)of reaching these nodes when it is identifying different routes throughthe virtual network between different source and destination endpoints.

FIG. 4C illustrates a routing graph 410 that is generated by adding twonodes 412 and 414 to the node graph 400 of FIG. 4A in order to representtwo SaaS providers. In this example, the known IP addresses are assignedto nodes 412 and 414, and these nodes are connected to nodes 402 and 404(representing two MFNs) through links 416 and 418 that have weights W1and W2 assigned to them. This approach is an alternative approach foradding the known IP addresses of the two SaaS providers to the approachillustrated in FIG. 4B.

FIG. 4D illustrates a more detailed routing graph 415. In this moredetailed routing graph, additional nodes 422 and 424 are added torepresent external corporate compute nodes (e.g., branch offices anddatacenters) with known IP addresses that connect respectively to theAWS and GCP public clouds 310 and 320. Each of these nodes 422/424 isconnected by at least one link 426 with an associated weight value Wi toat least one of the routing graph nodes that represents an MFN. Some ofthese nodes (e.g., some of the branch offices) are connected withmultiple links to same MFN or to different MFNs.

Next, at 525, the process 500 compute the lowest cost paths (e.g.,shortest paths, etc.) between each MFN and each other MFN that can serveas a virtual network egress location for a data message flow of thecorporate entity. The egress MFNs in some embodiments include the MFNsconnected to external corporate compute nodes (e.g., branch offices,corporate datacenters, and SaaS provider datacenters) as well as MFNsthat are candidate locations for mobile device connections and egressInternet connections. In some embodiments, this computation uses atraditional lowest-cost (e.g., shortest-path) identification processthat identifies the shortest paths between different MFN pairs.

For each candidate MFN pair, the lowest-cost identification process usesthe computed weight scores (i.e., the scores computed at 510) toidentify a path with the lowest score when multiple such paths existbetween the MFN pair. Several manners for computing lowest-cost pathswill be further described below. As mentioned above, thepath-identifying layer 282 identifies multiples paths between two MFNpairs in some embodiments. This is to allow the cloud forwardingelements 235 to use different paths under different circumstances.Accordingly, in these embodiments, the process 500 can identify multiplepaths between two MFN pairs.

At 530, the process removes from the routing graph the links between MFNpairs that are not used by any of the lowest-cost paths identified at525. Next, at 535, the process generates the routing tables for thecloud forwarding elements 235 from the routing graph. At 535, theprocess distributes these routing tables to the cloud forwardingelements 235 of the managed forwarding nodes. After 535, the processends.

In some embodiments, the virtual network has two types of externalconnections, which are: (1) external secure connections with the computenodes (e.g., branch offices, datacenters, mobile users, etc.) of anentity, and (2) external connections to third party computers (e.g.,SaaS provider servers) through the Internet. Some embodiments optimizethe virtual network by finding optimal virtual-network ingress andegress locations for each datapath that terminates at source anddestination nodes outside of the virtual network. For instance, toconnect a branch office to a SaaS provider server (e.g., salesforce.comserver), some embodiments connect the branch office to an optimal edgeMFN (e.g., the MFN that has the fastest network connection to the branchoffice or the one that is closest to the branch office), and identify anoptimal edge MFN to an optimally located SaaS provider server (e.g., theSaaS that is closest to the edge MFN for the branch office or has thefastest path to the edge MFN for the branch office through the edge MFNconnected to the SaaS provider server).

To associate each compute node (e.g., a branch office, a mobile user,etc.) of an entity to the closest MFN through a VPN connection, thevirtual network provider in some embodiments deploys one or moreauthoritative domain name servers (DNS) in the public clouds for thecompute nodes to contact. In some embodiments, each time a corporatecompute node in some embodiments needs to establish a VPN connection(i.e., to initialize or re-initialize the VPN connection) to an MFN ofthe virtual network provider, the compute node first resolves an addressassociated with its virtual network (e.g., virtualnetworkX.net) withthis authoritative DNS server in order to obtain from this server theidentity of the MFN that this server identifies as the MFN that isclosest to the corporate compute node. To identify this MFN, theauthoritative DNS server provides an MFN identifier (e.g., the IPaddress of the MFN) in some embodiments. The corporate compute node thenestablishes a VPN connection to this managed forwarding node.

In other embodiments, the corporate compute node does not first performa DNS resolution (i.e., does not first resolve a network address for aparticular domain) each time that it needs to establish a VPN connectionto an MFN of the VNP. For instance, in some embodiments, the corporatecompute node sticks with a DNS-resolved MFN for a particular duration(e.g., for a day, a week, etc.) before performing another DNS resolutionto determine whether this MFN is still an optimal one to which is shouldconnect.

When the source IP address in the DNS request is that of the local DNSserver of the corporate compute node, and not of the node itself, theauthoritative DNS server in some embodiments identifies the MFN closestto the local DNS server instead of the MFN closest to the corporatecompute node. To address this, the DNS request in some embodimentsidentifies the corporate compute node in terms of a domain name thatincludes one or more parts (labels) that are concatenated and delimitedby dots, where one of these parts identifies the corporation and theother part identifies the compute node of the corporation.

In some embodiments, this domain name specifies a hierarchy of domainsand sub-domains that descends from the right label to the left label inthe domain name. The right-most first label identifies the particulardomain, a second label to the left of the first label identifies thecorporate entity, and a third label to the left of the second labelidentifies the external machine location of the entity in cases wherethe entity has more than one external machine location. For instance, insome embodiments, the DNS request identifies the corporate compute nodeas myNode of company myCompany, and asks for the resolution of theaddress myNode.myCompany.virtualnetwork.net. The DNS server then usesthe myNode identifier to better select the ingress MFN to which thecorporate compute node should establish a VPN connection. In differentembodiments, the myNode identifier is expressed differently. Forexample, it may be addressed as an IP address, a latitude/longitudedescription of a location, a GPS (Global Positioning System) location, astreet address, etc.

Even when the IP address properly reflects the location, there may beseveral potential ingress routers, e.g., belonging to differentdatacenters in the same cloud or to different clouds in the same region.In such a case, the virtual network authoritative server in someembodiments sends back a list of IPs of potential MFN CFEs (e.g., C5,C8, C12). The corporate compute node in some embodiments then pings thedifferent CFEs in the list, to produce measurements (e.g., distance orspeed measurements), and selects the closest one by comparingmeasurements among the set of CFE candidates.

In addition, the corporate compute node may base this selection byidentifying the MFNs currently used by the other compute nodes of thecorporate entity. For example, in some embodiments, the corporatecompute node adds connection costs to each MFN, so that if many of thecorporate branches are already connected to a given cloud, new computenodes would have an incentive to connect to the same cloud, thusminimizing inter-cloud costs in terms of processing, latency, anddollars.

Other embodiments use other DNS resolution techniques. For instance,each time a corporate compute node (e.g., a branch office, datacenter, amobile user, etc.) needs to perform a DNS resolution, the corporatecompute node (e.g., the mobile device or a local DNS resolver at abranch office or datacenter) communicates with a DNS service providerthat serves as an authoritative DNS resolver for a number of entities.In some embodiments, this DNS service provider has DNS resolvingmachines located in one or more private datacenters, while in otherembodiments it is part of one or more public cloud datacenters.

To identify which of N managed forwarding nodes that connect directly tothe Internet should be used to reach a SaaS provider server, the virtualnetwork (e.g., the ingress MFN or the controller cluster that configuresthe MFNs) in some embodiments identifies a set of one or more candidateedge MFNs from the N managed forwarding nodes. As described furtherbelow, each candidate edge MFN in some embodiments is an edge MFN thatis deemed to be optimal based on a set of criteria, such as distance toSaaS provider server, network connection speed, cost, delay and/or loss,network compute cost, etc.

To assist in identifying the optimal edge points, the controller clusterof some embodiments maintains for an entity a list of the most popularSaaS providers and consumer web destinations and their IP addresssubnets. For each such destination, the controller cluster assigns oneor more of the optimal MFNs (again as judged by physical distance,network connection speed, cost, loss and/or delay, compute cost, etc.)as candidate egress nodes. For each candidate egress MFN, the controllercluster then computes the best route from each possible ingress MFN tothe candidate MFN, and sets up the resulting next-hop table in the MFNsaccordingly, such that the Internet SaaS provider or web destination isassociated to the correct virtual network next-hop node.

Given that the service destination can often be reached through severalIP subnets at several locations (as provided by the authoritative DNSserver), there are several potential egress nodes to minimize latencyand provide load-balancing. Accordingly, in some embodiments, thecontroller cluster computes the best location and egress node for eachMFN, and updates the next-hop accordingly. Also, the best egress node toget to a SaaS provider (e.g., office365.com) may be through one publiccloud provider (e.g., Microsoft Azure), but the best ingress MFN frompurely a distance or connection speed may be in another public cloudprovider (e.g., AWS). In such situations, it may not be optimal in termsof latency, processing and cost to traverse to another cloud (i.e., tothe public cloud with the best egress MFN) before leaving the virtualnetwork. Providing multiple candidate edge nodes would allow for theselection of an optimal edge MFN and an optimal path to the selectededge MFN in such situations.

To identify the optimal path through the virtual network to an egressMFN that connects to the Internet or connects to a corporate computenode of the corporate entity, the controller cluster identifies optimalrouting paths between the MFNs. As mentioned above, the controllercluster in some embodiments identifies the best path between any twoMFNs by first costing each link between a pair of directly connectedMFNs, e.g., based on a metric score that reflects the weighted sum ofestimated latency and financial costs. The latency and financial costsinclude in some embodiments (1) link delay measurements, (2) estimatedmessage processing latency, (3) cloud charges for outgoing traffic froma particular datacenter either to another datacenter of the same publiccloud provider, or to exit the public cloud (PC) provider's cloud (e.g.,to another public cloud datacenter of another public cloud provider orto the Internet), and (4) estimated message processing costs associatedwith the MFNs executing on host computers in the public clouds.

Using the computed costs of these pair-wise links, the controllercluster can compute the cost of each routing path that uses one or moreof these pair-wise links by aggregating the costs of the individualpair-wise links that are used by the routing path. As described above,the controller cluster then defines its routing graph based on thecomputed costs of the routing paths, and generates the forwarding tablesof the cloud routers of the MFNs based on the defined routing graphs.Also, as mentioned above, the controller cluster repeatedly performsthese costing, graph-building, and forwarding table update anddistribution operations periodically (e.g., once every 12 hours, 24hours, etc.) or as it receives measurement updates from the measurementagents of the MFNs.

Whenever the forwarding table at an MFN CFE C_(i) points to a next-hopMFN CFE C_(j), the CFE C_(i) considers C_(j) as a neighbor. In someembodiments, the CFE C_(i) establishes a secure, actively maintained VPNtunnel to CFE C_(j). A secure tunnel in some embodiments is a tunnelthat requires the payloads of the encapsulated data messages to beencrypted. Also, in some embodiments, a tunnel is actively maintained byone or both endpoints of the tunnel sending keep alive signals to theother endpoint.

In other embodiments, the CFEs do not establish secure, activelymaintained VPN tunnels. For instance, in some embodiments, the tunnelsbetween the CFEs are static tunnels that are not actively monitoredthrough the transmission of keep-alive signals. Also, in someembodiments, these tunnels between the CFEs do not encrypt theirpayloads. In some embodiments, the tunnels between pair of CFEs includetwo encapsulating headers, with the inner header identifying the tenantID and the ingress and egress CFEs for a data message entering andexiting the virtual network (i.e., entering and exiting the publiccloud(s)), and the outer encapsulating header specifying the source anddestination network addresses (e.g., IP addresses) for traversingthrough zero or more CFE from the ingress CFE to the egress CFE.

In addition to internal tunnels, the virtual network in some embodimentsconnects corporate compute nodes to their edge MFNs using VPN tunnels,as mentioned above. Therefore, in the embodiments where secure tunnelsare used to connect the CFEs, the data messages transit through virtualnetwork using an entirely secure VPN path.

As the virtual network data messages are forwarded using encapsulationwithin the virtual network, the virtual network in some embodiments usesits own unique network addresses that are different than the privateaddresses used by the different private networks of the tenant. In otherembodiments, the virtual network uses the private and public networkaddress spaces of the public clouds over which it is defined. In yetother embodiments, the virtual network uses some of its own uniquenetwork addresses for some of its components (e.g., some of its MFNs,CFEs, and/or services), while using the private and public networkaddress spaces of the public clouds for other of its components.

Also, in some embodiments, the virtual network uses a clean-slatecommunication platform with its own proprietary protocols. In theembodiments in which the data messages are forwarded entirely throughsoftware MFN routers (e.g., through software CFEs), the virtual networkcan provide an optimized rate control for long-haul end-to-endconnections. This is accomplished in some embodiments by operating a TCPoptimization proxy engine 220 at every MFN 150. In other embodimentsthat do not break the TCP itself (e.g., with HTTPS), this isaccomplished by the proxy engine 220 segmenting the rate control usingintermediate per-flow buffering together with TCP receiver-window andACK manipulation.

Due to its clean-slate nature, the virtual network in some embodimentsoptimizes many of its components to provide an even better service. Forinstance, in some embodiments, the virtual network uses multiple-pathrouting to support premium bandwidth-guaranteed VPN setups that arerouted across the virtual network. In some embodiments, such VPNsinclude state data in each MFN similar to ATM/MPLS routing, and theirestablishment and removal is centrally controlled. Some embodimentsidentify the available bandwidth per outgoing link, either by measuringit directly (through packet pair or a similar process) or by having agiven capacity for the link and reducing from this capacity the trafficthat is already sent through this link.

Some embodiments use the residual bandwidth of a link as a constraint.For instance, when a link does not have at least 2 Mbps of availablebandwidth, the controller cluster of some embodiments removes the linkfrom the set of links that are used to compute lowest-cost path (e.g.,shortest path) to any destination (e.g., remove the link from therouting graph, such as graph 400). If an end-to-end route is stillavailable after the removal of this link, new VPNs will be routed acrossthis new route. VPN removal can bring back available capacity to a givenlink, which in turn can enable this link to be included in thelowest-cost path (e.g., shortest path) calculation. Some embodiments useother options for multiple-path routing such as load balancing oftraffic across multiple paths, e.g., using MPTCP (multi-path TCP).

Some embodiments provide a better service for premium customers byexploiting the path parallelism and the inexpensive cloud links toduplicate traffic from the ingress MFNs to the egress MFN, through twodisjoint paths (e.g., maximally disjoint paths) within the virtualnetwork. Under this approach, the earliest message that arrives isaccepted, and the later one discarded. This approach increases thevirtual network reliability and reduces the delay, at the cost ofincreasing the egress processing complexity. In some such embodiments,Forward Error Correction (FEC) techniques are used to increasereliability while reducing the duplication traffic. Due to itsclean-slate nature, the virtual network of some embodiments performsother upper-layer optimizations, such as application-layer optimizations(e.g., de-duplication and caching operations) and security optimizations(e.g., the addition of encryption, DPI (deep packet inspection) andfirewalling).

The virtual network of some embodiments accounts for collaboration withcloud providers, to further improve the virtual network setup by usinganycast messaging. For instance, in some embodiments when all MFNsobtain the same external IP address, it is easier to connect any newcorporate compute node to an optimal edge node (e.g., the closest edgenode) using an anycast connection. Likewise, any SaaS provider canobtain this IP address and connect to the optimal MFN (e.g., closestMFN).

As mentioned above, different embodiments use different types of VPNconnections to connect corporate compute nodes (e.g., branches andmobile devices) to the MFNs that establish the virtual network of acorporate entity. Some embodiments use IPsec to set up these VPNconnections. FIG. 6 illustrates the IPsec data message format of someembodiments. Specifically, this figure illustrates an original format ofa data message 605 generated by a machine at the corporate compute node,and an IPsec encapsulated data message 610 after the data message 605has been encapsulated (e.g., at the corporate compute node or the MFN)for transmission through an IPsec tunnel (e.g., to the MFN or to thecorporate compute node).

In this example, the IPsec tunnel is set up with ESP Tunnel Mode, port50. As shown, this mode is set up in this example by replacing the TCPprotocol identifier in the IP header with an ESP protocol identifier.The ESP header identifies the start of the message 615 (i.e., the header620 and payload 625). The message 615 has to be authenticated by therecipient of the IPsec encapsulated data message (e.g., by the IPsecgateway of the MFN). The start of the payload 625 is identified by thevalue of the next field 622 of the message 615. Also, the payload 625 isencrypted. This payload includes the IP header, the TCP header andpayload of the original data message 605, as well as a padding field630, which includes the next field 622.

In some embodiments, each MFN IPsec gateway can handle multiple IPsecconnections for the same or different virtual network tenants (e.g., forthe same corporation or for different corporations). Accordingly, an MFNIPsec gateway (e.g., gateway 230) in some embodiments identifies eachIPsec connection in terms of a tunnel ID, a tenant ID (TID), and acorporate compute node subnet. In some embodiments, different corporatenodes (e.g., different branch offices) of a tenant do not haveoverlapping IP subnets (per RFC 1579). The IPsec gateway in someembodiments has a table mapping each IPsec tunnel ID (which is containedin the IPsec tunnel header) to a tenant ID. For a given tenant that anIPsec gateway is configured to handle, the IPsec gateway also has amapping of all subnets of that tenant that connect to the virtualnetwork established by the MFNs and their cloud forwarding elements.

When an ingress first MFN in a first public cloud datacenter receivesthrough an IPsec tunnel a data message associated with a tenant ID anddestined to a destination (e.g., a branch or datacenter subnet, or aSaaS provider) that connects to an egress second MFN in a second publiccloud datacenter, the IPsec gateway of the first MFN removes the IPsectunnel header. In some embodiments, the CFE of the first MFN thenencapsulates the message with two encapsulating headers that allow themessage to traverse a path from the ingress first MFN to the egresssecond MFN, directly or through one or more other intermediate MFNs. TheCFE of the first MFN identifies this path by using itscontroller-configured routing table.

As mentioned above, the two encapsulating headers in some embodimentsinclude (1) an outer header that specifies the next hop MFN CFE to allowthe encapsulated data message to traverse through the MFNs of thevirtual network to reach the egress MFN CFE, and (2) an inner headerthat specifies the tenant ID and the ingress and egress MFN CFEs thatidentify the MFNs for the data message entering and exiting the virtualnetwork.

Specifically, in some embodiments, the inner encapsulating headerincludes a valid IP header with the destination IP address of the egresssecond MFN's CFE and the source IP address of the ingress first MFN'sCFE. This approach allows standard IP router software to be used inevery CFE of the MFNs. The encapsulation further includes the tenant ID(e.g., a customer CID). When a message arrives at the egress secondMFN's CFE, it is decapsulated and sent by the second MFN to itsdestination (e.g., sent by the second MFN's IPsec gateway to thedestination via another IPsec tunnel that is associated with the tenantID and the destination subnet of the message).

Certain cloud providers prohibit machines from “spoofing” source IP,and/or impose other restrictions for TCP and UDP traffic. To deal withsuch possible restrictions, some embodiments use the outer header toconnect neighboring pairs of MFNs that are used by one or more routes.This header in some embodiments is a UDP header that specifies sourceand destination IP addresses and the UDP protocol parameters. In someembodiments, the ingress MFN CFE specifies its IP address as the sourceIP address of the outer header, while specifying the next MFN CFE hop'sIP address as the destination IP address of the outer header.

When the path to the egress MFN's CFE includes one or more intermediateMFN CFEs, an intermediate CFE replaces the source IP address in theouter header of the double-encapsulated message that it receives withits IP address. It also uses the destination IP address in the innerheader to perform a route lookup in its routing table to identify thedestination IP address of the next hop MFN CFE that is on the path tothe destination IP address of the inner header. The intermediate CFEthen replaces the destination IP address in the outer header with the IPaddress that it identified through its route table lookup.

When the double encapsulated data message reaches the egress MFN's CFE,the CFE determines that it is the egress node for the data message whenit retrieves the destination IP address in the inner header anddetermines that this destination IP address belongs to it. This CFE thenremoves the two encapsulating headers from the data message and thensends it to it destination (e.g., through its MFN's IPsec gateway to thedestination via another IPsec tunnel that is associated with the tenantID and the destination IP address or subnet in the data message'soriginal header).

FIG. 7 illustrates an example of the two encapsulating headers of someembodiments, while FIG. 8 presents an example that illustrates how thesetwo headers are used in some embodiments. In the discussion below, theinner header is referred to as the tenant header as it includes thetenant ID along with the identity of the virtual-network ingress/egressnodes connected to the tenant's corporate compute end nodes. The outerheader is referred to below as the VN-hop tunnel header because it isused to identify the next hop through the virtual network as the datamessage traverses a path through the virtual network between ingress andegress MFN CFEs.

FIG. 7 shows a VN-hop tunnel header 705 and a tenant tunnel header 720encapsulating an original data message 750 with an original header 755and a payload 760. As shown, the VN-hop tunnel header 705 in someembodiments includes a UDP header 710 and an IP header 715. The UDPheader in some embodiments is defined according to a UDP protocol. Insome embodiments, the VN-hop tunnel is a standard UDP tunnel, while inother embodiments, this tunnel is a proprietary UDP tunnel. In stillother embodiments, this tunnel is a standard or proprietary TCP tunnel.The tunnel header 705 in some embodiments is an encrypted one thatencrypts its payload, while in other embodiments it is an unencryptedtunnel.

As further described below, the tunnel header 705 in some embodiments isused to define an overlay VNP network, and is used by each MFN CFE toreach the next hop MFN CFE over the underlay public cloud networks. Assuch, the IP header 715 of the tunnel header 705 identifies the sourceand destination IP addresses of the first and second CFEs of the firstand second neighboring MFNs connected by the VNP tunnel. In some cases(e.g., when the next hop destination MFN is in a different public cloudof a different public cloud vendor than the source MFN), the source anddestination IP addresses are public IP addresses that are used by thepublic cloud datacenters that include the MFNs. In other cases, when thesource and destination MFN CFEs belong to the same public cloud, thesource and destination IP addresses can be private IP addresses that areused in just the public cloud. Alternatively, in such cases, the sourceand destination IP addresses might still be public IP addresses of thepublic cloud vendor.

As shown in FIG. 7, the tenant tunnel header 720 includes an IP header725, a tenant ID field 730 and a virtual circuit label (VCL) 735. Thetenant tunnel header 720 is used by each hop CFE after the ingress hopCFE to identify the next hop for forwarding the data message to theegress CFE of the egress MFN. As such, the IP header 725 includes asource IP address that is the IP address of the ingress CFE and adestination IP address that is the IP address of the egress CFE. As withthe source and destination IP addresses of the VN-hop header 705, thesource and destination IP addresses of the tenant header 720 can beeither private IP addresses of one public cloud provider (when the datamessage traverses a route that only goes through one public cloudprovider's datacenter), or public IP addresses of one or more publiccloud providers (e.g., when the data message traverses a route that goesthrough datacenters of two or more public cloud providers).

The IP header of the tenant header 720 can be routed by using anystandard software router and IP routing table in some embodiments. Thetenant ID field 730 contains the tenant ID, which is a unique tenantidentifier that can be used at the ingress and egress MFNs to uniquelyidentify a tenant. The virtual network provider in some embodimentsdefines different tenant IDs for different corporate entities that aretenants of the provider. The VCL field 735 is an optional routing fieldthat some embodiments use to provide an alternative way (non-IP basedway) for forwarding messages through the network. In some embodiments,the tenant tunnel header 720 is a GUE (Generic UDP Encapsulation)header.

FIG. 8 presents an example that illustrates how these two tunnel headers705 and 710 are used in some embodiments. In this example, a datamessages 800 is sent from a first machine 802 (e.g., first VM) in afirst branch office 805 of a company to a second machine 804 (e.g., asecond VM) in a second branch office 810 of the company. The twomachines are in two different subnets, which are 10.1.0.0 and 10.2.0.0,with the first machine having an IP address 10.1.0.17 and the secondmachine having an IP address 10.2.0.22. In this example, the firstbranch 805 connects to an ingress MFN 850 in a first public clouddatacenter 830, while the second branch 810 connects to an egress MFN855 in a second public cloud datacenter 838. Also, in this example, theingress and egress MFNs 850 and 855 of the first and second public clouddatacenters are indirectly connected through an intermediate MFN 857 ofa third public cloud datacenter 836.

As shown, the data message 800 from machine 802 is sent to the ingressMFN 850 along an IPsec tunnel 870 that connects the first branch office805 to the ingress MFN 850. This IPsec tunnel 870 is established betweenan IPsec gateway 848 of the first branch office and an IPsec gateway 852of the ingress MFN 850. This tunnel is established by encapsulating thedata message 800 with an IPsec tunnel header 806.

The IPsec gateway 852 of the MFN 850 decapsulates the data message(i.e., removes the IPsec tunnel header 806), and passes the decapsulatedmessage to this MFN's CFE 832 directly or through one or more middleboxservice machines (e.g., through a firewall machine, such as machine 210of FIG. 2). In passing this message, the IPsec gateway or some othermodule of the MFN 850 in some embodiments associates the message withthe tunnel ID of the IPsec tunnel and a tenant ID of the company. Thistenant ID identifies the company in the records of the virtual networkprovider.

Based on the associated tenant ID and/or the IPsec tunnel ID, the CFE832 of the ingress MFN 850 identifies a route for the message to itsdestination machine's subnet (i.e., to the second branch office 810)through the virtual network that is established by the MFNs in thedifferent public cloud datacenters. For instance, the CFE 832 uses thetenant ID and/or the IPsec tunnel ID to identify the routing table forthe company. In this routing table, the CFE 832 then uses thedestination IP address 10.2.0.22 of the received message to identify arecord that identifies the CFE 853 of the egress MFN 855 of the publiccloud datacenter 838 as the destination egress forwarding node for thedata message 800. In some embodiments, the identified record maps theentire subnet 10.2.0.0/16 of the second branch office 810 to the CFE 853of the MFN 855.

After identifying the egress CFE 853, the CFE 832 of the ingress MFN 850encapsulates the received data message with a tenant tunnel header 860that in its IP header 725 includes the source IP of the ingress CFE 832and the destination IP of the egress CFE 853. In some embodiments, theseIP addresses are defined in the public IP address space. The tunnelheader 860 also includes the tenant ID that was associated with the datamessage at ingress MFN 850. As mentioned above, this tunnel header alsoincludes the VCL header value in some embodiments.

In some embodiments, the ingress CFE 832 also identifies the next hopMFN that is on the desired CFE routing path to the egress CFE 853. Insome embodiments, the ingress CFE 832 identifies this next hop CFE inits routing table by using the destination IP address of the egress CFE853. The next hop MFN CFE in this example is the CFE 856 of the thirdMFN 857 of a third public cloud datacenter 836.

After identifying the next hop MFN CFE, the ingress MFN CFE encapsulatesthe encapsulated data message 800 with a VN-hop, second tunnel header862. This tunnel header allows the message to route to the next hop CFE856. In the IP header 715 of this outer header 862, ingress MFN CFE 832specifies the source and destination IP addresses as the source IP ofthe ingress CFE 832 and the destination IP of the intermediate CFE 856.It also specifies its layer 4 protocol as being UDP in some embodiments.

When the CFE 856 of the third MFN 857 receives the double-encapsulateddata message, it removes the VN-hop, second tunnel header 862, and theextracts from the tenant header 860 the destination IP address of theCFE 853 of the egress MFN 855. Since this IP address is not associatedwith the CFE 856, the data message still has to traverse to another MFNto reach its destination. Accordingly, the CFE 856 uses the extracteddestination IP address to identify a record in its routing table thatidentifies the next hop MFN CFE 853. It then changes re-encapsulates thedata message with the outer header 705 and specifies the source anddestination IP addresses in its IP header 715 as its own IP address andthe destination IP address of the MFN CFE 853. Next, the CFE 856forwards the double-encapsulated data message 800 to the egress CFE 853through intervening routing fabric of the public cloud datacenters 836and 838.

After receiving the encapsulated data message, the egress CFE 853determines that the encapsulated message is directed to it when itretrieves the destination IP address in the inner header 860 anddetermines that this destination IP address belongs to it. The egressCFE 853 removes both encapsulating headers 860 and 862 from the datamessage 800, and extracts the destination IP address in the datamessage's original header. This destination IP address identifies the IPaddress of the second machine 804 in the second branch office's subnet.

Using the tenant ID in the removed tenant tunnel header 860, the egressCFE 853 identifies the correct routing table to search, and thensearches this routing table based on the destination IP addressextracted from the original header value of the received data message.From this search, the egress CFE 853 identifies a record that identifiesthe IPsec connection to use to forward the data message to itsdestination. It then provides the data message along with the IPsecconnection identifier to the second MFN's IPsec gateway 858, which thenencapsulates this message with an IPsec tunnel header 859 and thenforwards it to an IPsec gateway 854 of the second branch office 810. Thegateway 854 then removes the IPsec tunnel header and forwards the datamessage to its destination machine 804.

Several more detailed message-processing examples will now be describedby reference to FIGS. 9-15. In these examples, it is assumed that eachtenant IPsec interface is on the same local public IP address, as arethe VNP tunnels. As such, the interfaces in some embodiments areattached to a single VRF (virtual routing and forwarding) namespace.This VRF namespace is referred to below as the VNP namespace.

FIGS. 9-11 illustrate message-handling processes 900-1100 that areperformed respectively by the ingress, intermediate, and egress MFNswhen they receive a message that is sent between two compute devices intwo different external machine locations (e.g., branch offices,datacenters, etc.) of a tenant. In some embodiments, the controllercluster 160 configures the CFE of each MFN to operate as an ingress,intermediate, and egress CFE, when each such CFE is a candidate to serveas an ingress, intermediate and egress CFE for different data messageflows of a tenant.

The processes 900-1100 will be explained below by reference to twoexamples in FIGS. 8 and 12. As mentioned above, FIG. 8 illustrates anexample when the data message goes through an intermediate MFN to get tothe egress MFN. FIG. 12 illustrates an example that does not involve anintermediate MFN between the ingress and egress MFNs. Specifically, itillustrates a data message 1200 being sent from a first device 1202 in afirst branch office 1205 to a second device 1210 in a second branchoffice 1220 when the two branch offices connect to two public clouddatacenters 1230 and 1238 with two MFNs 1250 and 1255 that are directlyconnected. As shown, the CFEs 1232 and 1253 of the MFNs in theseexamples perform the routing operations associated with each MFN.

The ingress CFE (e.g., ingress CFE 832 or 1232) of the ingress MFNs 850and 1250 perform the process 900 in some embodiments. As shown in FIG.9, the ingress process 900 starts by initially identifying (at 905) thetenant routing context based on the identifier of the IPsec tunnel(e.g., 806 or 1206) in the received data message. In some embodiments,the IPsec gateways or other MFN modules store the tenant IDs for theIPsec tunnel IDs in mapping tables. Whenever a data message is receivedalong a particular IPsec tunnel, the IPsec gateway extracts the IPsectunnel ID, which this gateway or another MFN module then uses toidentify the associated tenant ID by reference to its mapping table. Byidentifying the tenant ID, the process identifies the tenant routingtable or the tenant portion of the VRF namespace to use.

At 910, the process increments the identified IPsec tunnel's RX(receive) counter to account for receiving this data message. Next, at915, the process performs a route lookup (e.g., a longest prefix match,LPM, lookup) in the identified tenant routing context (e.g., in thetenant's portion of the VRF namespace) to identify the IP address of theegress interface for exiting the tenant's virtual network that is builtover the public cloud datacenters. For the branch-to-branch examples,the egress interface is the IP address of an egress CFE (e.g., CFE 853or 1253) of an MFN connected to the destination branch.

At 920, the process adds a tenant tunnel header (e.g., header 860 or1260) to the received data message, and embeds the source IP address ofthe ingress CFE (e.g., ingress CFE 832 or 1252) and the destination IPaddress of the egress CFE (e.g., egress CFE 853 or 1253) as the sourceand destination IP addresses in this tunnel header. In the tenantheader, the process also stores the tenant ID (identified at 905) in thetenant header. At 920, the process adds a VN-hop tunnel header (e.g.,header 862 or 1262) outside of the tenant header, and stores its IPaddress as the source IP address in this header. The process alsospecifies (at 920) the UDP parameters (e.g., UDP port) of the VNP tunnelheader.

Next, at 925, the process increments the VN-transmit counter for thetenant to account for this data message's transmission. At 930, theprocess performs a route lookup (e.g., an LPM lookup) in the identifiedVNP routing context (e.g., in the VNP's portion of the VRF namespace) toidentify the next hop interface for this data message. In someembodiments, this route lookup is an LPM lookup (e.g., in the VNP'sportion of the VRF namespace) that is at least partially based on theegress CFE's destination IP.

At 935, the process determines whether the next hop egress interface isa local interface (e.g., a physical or virtual port) of the ingress CFE.If so, the process defines (at 937) the destination IP address in theVN-hop outer tunnel header as the egress interface IP address identifiedat 915. Next, at 940, the process provides the double encapsulated datamessage to its local interface so that it can be forwarded to thedestination egress CFE. After 940, the process 900 ends.

FIG. 12 illustrates an example of the operation 905-940 for the datamessage 1200 that the ingress CFE 1232 receives from the device 1202 ofthe first branch office 1205. As shown, this CFE's MFN 1250 receivesthis data message as an IPsec encapsulated message at its IPsec gateway1252 from the IPsec gateway 1248 of the first branch office 1205. Theingress CFE 1232 encapsulates the received message 1200 (after its IPsecheader has been removed by an IPsec gateway 1252) with a VN-hop tunnelheader 1262 and a tenant tunnel header 1260, and forwards this doubleencapsulated message to the egress CFE 1253 of MFN 1255 of public cloud1238. As shown, the source and destination IP addresses of both tunnelheaders 1260 and 1262 are identical in this example. Given that thesetwo sets of IP addresses are identical, some embodiments forego usingthe outer IP header 1262 when the data message is not routed through anyintervening CFE, such as CFE 856.

When the process determines (at 935) that the next hop egress interfaceis not a local interface of the ingress CFE but rather is thedestination IP address of another router, the process embeds (at 945) inthe VN-hop tunnel header, the destination IP address of the next hopintermediate CFE (e.g., intermediate CFE 856) as the destination IPaddress of the VN-hop tunnel header.

Next, at 950, the process performs another route lookup (e.g., an LPMlookup) in the identified VNP routing context (e.g., in the VNP'sportion of the VRF namespace). This time, the lookup is based on the IPaddress of the intermediate CFE that is identified in the VNP tunnelheader. As the intermediate CFE (e.g., CFE 856) is a next-hop CFE in thevirtual network for the ingress CFE (e.g., CFE 832), the routing tableidentifies a local interface (e.g., a local port) for data messages sentto the intermediate CFE. Thus, this lookup in the VNP routing contextidentifies a local interface, to which the ingress CFE provides (at 950)the double-encapsulated message. The process then increments (at 955)the VN-intermediate counter to account for this data message'stransmission. After 955, the process ends.

FIG. 10 illustrates a process 1000 that a CFE (e.g., CFE 853 or 1253) ofan egress MFN performs in some embodiments when it receives a datamessage that should be forwarded to a corporate compute node (e.g., abranch office, datacenter, remote user location) connected to the MFN.As shown, the process initially receives (at 1005) the data message onan interface associated with the virtual network. This message isencapsulated with the VN-hop tunnel header (e.g., header 862 or 1262)and tenant tunnel header (e.g., header 860 or 1260).

At 1010, the process determines that the destination IP address in theVN-hop tunnel header is its CFE's destination IP address (e.g., IPaddress of CFE 853 or 1253). Next, at 1015, the process removed the twotunnel headers. The process then retrieves (at 1020) the tenant ID fromthe removed tenant tunnel header. To account for the received datamessage, the CFE then increments (at 1025) the RX (receive) counter thatit maintains for the tenant specified by the extracted tenant ID.

Next, at 1030, the process performs a route lookup (e.g., an LPM lookup)in the identified tenant routing context (i.e., in the routing contextof the tenant identified by the tenant ID extracted at 1020) to identifythe next hop interface for this data message. The process performs thislookup based on the destination IP address in the original header (e.g.,header 755) of the received data message in some embodiments. From therecord identified through this lookup, the process 1000 identifies theIPsec interface through which the data message has to be sent to itsdestination. Accordingly, the process 1000 sends the decapsulated,received data message to its MFN's IPsec gateway (e.g., gateway 858 or1258).

This gateway then encapsulates the data message with an IPsec tunnelheader (e.g., tunnel header 859 or 1259) and forwards it to a gateway(e.g., gateway 854 or 1254) in the destination corporate compute node(e.g., destination branch office), where it will be decapsulated andforwarded to its destination. After 1030, the CFE or its MFN increments(at 1035) the counter that it maintains for transmitting messages alongthe IPsec connection to the destination corporate compute node (e.g.,the IPsec connection between gateways 854 and 858, or between gateways1254 and 1258).

FIG. 11 illustrates a process 1100 that a CFE (e.g., CFE 856) of anintermediate MFN performs in some embodiments when it receives a datamessage that should be forwarded to another CFE of another MFN. Asshown, the process initially receives (at 1105) the data message on aninterface associated with the virtual network. In some embodiments, thismessage is encapsulated with two tunnel headers, a VN-tunnel header(e.g., header 862) and a tenant tunnel header (e.g., header 860).

At 1110, the process terminates the VN-hop tunnel as it determines thatthe destination IP address in this tunnel header is its CFE'sdestination IP address (e.g., is the destination IP address of CFE 856).Next, at 1115, the process determines whether the VN-hop tunnel headerspecifies the correct UDP port. If not, the process ends. Otherwise, at1120, the process removes the VN-hop tunnel header. To account for thereceived data message, the CFE then increments (at 1125) the RX(receive) counter that it maintains to quantify the number of messagesthat it has received as an intermediate hop CFE.

At 1130, the process performs a route lookup (e.g., an LPM lookup) inthe identified VNP routing context (e.g., in the VNP's portion of theVRF namespace) to identify the next hop interface for this data message.In some embodiments, this route lookup is an LPM lookup (e.g., in theVNP's portion of the VRF namespace) that is at least partially based onthe egress CFE's destination IP that is identified in the inner tenanttunnel header.

The process then determines (at 1135) whether the next hop egressinterface is a local interface of the intermediate CFE. If so, theprocess adds (at 1140) the VN-hop tunnel header to the data message,which is already encapsulated with the tenant tunnel header. The processsets (at 1142) the destination IP address in the VN-hop tunnel header tothe egress CFE's destination IP address that is specified in the tenanttunnel header. It also sets (at 1142) the source IP address in theVN-hop tunnel header to the IP address of its CFE. In this tunnelheader, the process also sets the UDP attributes (e.g., the UDP port,etc.).

Next, at 1144, the process provides the double encapsulated data messageto its local interface (identified at 1130) so that it can be forwardedto the destination egress CFE. One example of this VN-hop tunnelde-capsulation and forwarding was described above by reference to theoperations of CFE 856 in FIG. 8. To account for the received datamessage, the CFE then increments (at 1146) the TX (transmit) counterthat it maintains to quantify the number of messages that it hastransmitted as an intermediate hop CFE. After 1146, the process 1100ends.

On the other hand, when the process determines (at 1135) that the nexthop egress interface is not a local interface of its CFE but rather isthe destination IP address of another router, the process adds (at 1150)a VN-hop tunnel header to the data message from which it previouslyremoved a VN-hop tunnel header. In the new VN-hop tunnel header, theprocess 1100 embeds (at 1150) the source IP address of its CFE and thedestination IP address (identified at 1130) of the next hop intermediateCFE as the source and destination IP addresses of the VN-hop tunnelheader. This VNP tunnel header also specifies a UDP layer 4 protocolwith a UDP destination port.

Next, at 1155, the process performs another route lookup (e.g., an LPMlookup) in the identified VNP routing context (e.g., in the VNP'sportion of the VRF namespace). This time, the lookup is based on the IPaddress of the next hop intermediate CFE that is identified in the newVN-hop tunnel header. As this intermediate CFE is a next-hop of thecurrent intermediate CFE in the virtual network, the routing tableidentifies a local interface for data messages sent to the next-hopintermediate CFE. Thus, this lookup in the VNP routing contextidentifies a local interface, to which the current intermediate CFEprovides the double-encapsulated message. The process then increments(at 1160) the VN-intermediate TX (transmit) counter to account for thisdata message's transmission. After 1160, the process ends.

FIG. 13 illustrates a message-handling process 1300 that is performed bythe CFE of the ingress MFN when it receives a message for a tenant thatis sent from a corporate compute device of the tenant (e.g., in a branchoffice) to another tenant machine (e.g., in another branch office,tenant datacenter or a SaaS provider datacenter). The process 900 ofFIG. 9 is a subset of this process 1300 as further described below. Asshown in FIG. 13, the process 1300 starts by initially identifying (at905) the tenant routing context based on the identifier of the incomingIPsec tunnel.

At 1310, the process determines whether both the source and destinationIP addresses in the received data message's header are public IPaddresses. If so, the process (at 1315) drops the data message andincrements the drop counter that it maintains for the received datamessage's IPsec tunnel. At 1315, the process drops the counter becauseit should not be receiving messages that are addressed to and frompublic IP addresses when it receives the messages through the tenant'sIPsec tunnel. In some embodiments, the process 1300 also sends back tothe source corporate compute machine an ICMP error message.

On the other hand, when the process determines (at 1310) that the datamessage is not coming from a public IP address and going to anotherpublic IP address, the process determines (at 1320) whether thedestination IP address in the received data message's header is a publicIP address. If so, the process transitions to 1325 to perform process900 of FIG. 9, with the exception of operation 905, which it hasperformed at the start of the process 1300. After 1325, the process 1300ends. On the other hand, when the process 1300 determines (at 1320) thatthe destination IP address in the received data message's header is nota public IP address, the process increments (at 1330) the identifiedIPsec tunnel's RX (receive) counter to account for receiving this datamessage.

The process 1300 then performs (at 1335) a route lookup (e.g., an LPMlookup) in the identified tenant routing context (e.g., in the tenant'sportion of the VRF namespace). This lookup identifies the IP address ofthe egress interface for exiting the tenant's virtual network that isbuilt over the public cloud datacenters. In the example illustrated inFIG. 13, the process 1300 reaches the lookup operation 1335 when thedata message is intended for a machine in a SaaS provider datacenter.Hence, this lookup identifies the IP address of the egress router forexiting the tenant's virtual network to reach the SaaS provider machine.In some embodiments, all the SaaS provider routes are installed in oneroute table or in one portion of the VRF namespace, while in otherembodiments the routes for the different SaaS providers are stored indifferent route tables or different VRF namespace portions.

At 1340, the process adds a tenant tunnel header to the received datamessage, and embeds the source IP address of the ingress CFE and thedestination IP address of the egress router as the source anddestination IP addresses in this tunnel header. Next, at 1345, theprocess increments the VN-transmit counter for the tenant to account forthis data message's transmission. At 1350, the process performs a routelookup (e.g., an LPM lookup) in the VNP routing context (e.g., in theVNP's portion of the VRF namespace) to identify one of its localinterfaces as the next hop interface for this data message. When thenext hop is another CFE (e.g., in other public cloud datacenter), theprocess in some embodiments further encapsulates the data message withthe VN-hop header, and embeds its CFE's IP address and the other CFE'sIP address as the source and destination addresses of the VN-hop header.At 1355, the process provides the encapsulated data message to itsidentified local interface so that the data message can be forwarded toits egress router. After 1355, the process 1300 ends.

In some cases, the ingress MFN can receive a data message for a tenantthat its CFE can directly forward to the data message's destinationmachine without going through another MFN's CFE. In some such cases, thedata message does not need to be encapsulated with a tenant header or aVN-hop header when the CFE does not need to relay any tenant specificinformation to any other subsequent VN processing module or the neededinformation can be provided to the subsequent VN processing modulethrough other mechanisms.

For instance, to directly forward a tenant's data message to an externalSaaS provider datacenter, the ingress MFN's NAT engine 215 would have toperform a NAT operation based on the tenant identifier, as furtherdescribed below. The ingress CFE or another module in the ingress MFNhas to provide the tenant identifier to the ingress MFN's associated NATengine 215. When the ingress CFE and NAT engines execute on the samecomputer, some embodiments share this information between these twomodules by storing it in a shared memory location. On the other hand,when the CFE and NAT engines do not execute on the same computer, someembodiments use other mechanisms (e.g., an out-of-band communication) toshare the tenant ID between the ingress CFE and NAT engines. In suchcases, however, other embodiments use an encapsulating header (i.e., usean in-band communication) to store and share the tenant ID betweendifferent modules of the ingress MFN.

In some embodiments, a virtual network provider uses the above-describedprocesses, systems, and components to provide multiple virtual WANs formultiple different tenants (e.g., multiple different corporate WANs formultiple corporations) over multiple public clouds of the same ordifferent public cloud providers. FIG. 14 presents an example that showsM virtual corporate WANs 1415 for M tenants of a virtual networkprovider that has network infrastructure and controller cluster(s) 1410in N public clouds 1405 of one or more public cloud providers.

Each tenant's virtual WAN 1415 can span all of the N public clouds 1405,or a subset of these public clouds. Each tenant's virtual WAN 1415connects one or more branch offices 1420, datacenters 1425, SaaSprovider datacenters 1430, and remote devices of the tenant. In someembodiments, each tenant's virtual WAN spans any public cloud 1405 thatthe VNP's controller cluster deems necessary for efficiently forwardingdata messages between the different compute nodes 1420-1435 of thetenant. In selecting the public clouds, the controller cluster in someembodiments also accounts for public clouds that the tenant selectsand/or the public clouds in which the tenant, or at least one SaaSprovider of the tenant, has one or more machines.

The virtual WAN 1415 of each tenant allows the remote devices 1435(e.g., mobile devices or remote computers) of the tenant to avoidinteracting with the tenant's WAN gateway at any branch office or tenantdatacenter, in order to access a SaaS provider service (i.e., to accessa SaaS provider machine or machine cluster). The tenant's virtual WAN insome embodiments allows the remote devices to avoid the WAN gateways atthe branch offices and tenant datacenters, by moving the functionalitiesof these WAN gateways (e.g., the WAN security gateways) to one or moremachines in the public clouds spanned by the virtual WAN.

For example, to allow a remote device to access the compute resources ofthe tenant or its SaaS provider services, a WAN gateway in someembodiments has to enforce firewall rules that control how the remotedevice can access the tenant's computer resources or its SaaS providerservices. To avoid branch or datacenter WAN gateways of the tenant, thetenant's firewall engines 210 are placed in the virtual network MFNs inone or more public clouds spanned by the tenant's virtual WAN.

The firewall engines 210 in these MFNs perform the firewall serviceoperations on the data message flows from and to the remote devices. Byperforming these operations in the virtual network deployed over one ormore public clouds, the data message traffic associated with thetenant's remote devices do not need to be unnecessarily routed throughthe tenant's datacenter(s) or branch offices in order to receivefirewall rule processing. This alleviates traffic congestion in thetenant datacenters and branch offices, and avoids consuming expensiveingress/egress network bandwidth at these locations for processingtraffic that is not destined to compute resources at these locations. Italso helps speed up the forwarding of the data message traffic from andto the remote devices as this approach allows the intervening firewallrule processing to occur within the virtual network as the data messageflows traverse to their destinations (e.g., at their ingress MFNs,egress MFNs or intermediate-hop MFNs).

In some embodiments, the firewall enforcing engine 210 (e.g., firewallservice VM) of an MFN receives firewall rules form the VNP centralcontrollers 160. A firewall rule in some embodiments includes a ruleidentifier and an action. The rule identifier in some embodimentsincludes one or more match values that are to be compared to datamessage attributes, such as layer 2 attributes (e.g., MAC addresses),layer 3 attributes (e.g., five tuple identifiers, etc.), tenant ID,location ID (e.g., office location ID, datacenter ID, remote user ID,etc.), in order to determine whether the firewall rule matches a datamessage.

The firewall rule's action in some embodiments specifies the action(e.g., allow, drop, re-direct, etc.) that the firewall enforcing engine210 has to take on a data message when the firewall rule matches thedata message's attributes. To address the possibility that multiplefirewall rules match a data message, the firewall enforcing engine 210stores the firewall rules (that it receives from the controller cluster160) in a firewall rule data storage in a hierarchical manner so thatone firewall rule can have higher priority than another firewall rule.When a data message matches two firewall rules, the firewall enforcingengine applies the rule with the higher priority in some embodiments. Inother embodiments, the firewall enforcing engine examines the firewallrules according to their hierarchy (i.e., examines higher priority rulesbefore lower priority rules) in order to ensure that it first matchesthe higher priority rule in case another lower priority rule might alsobe a match for the data message.

Some embodiments allow the controller cluster to configure the MFNcomponents to have the firewall service engines examine a data messageat an ingress node (e.g., node 850) as it enters a virtual network, atan intermediate node (e.g., node 857) on the virtual network or at anegress node (e.g., node 855) as it exits the virtual network. At each ofthese nodes, the CFE (e.g., 832, 856, or 858) in some embodiments callsits associated firewall service engine 210 to perform the firewallservice operation on the data message that the CFE receives. In someembodiments, the firewall service engine returns its decision to themodule that called it (e.g., to the CFE) so that this module can performthe firewall action on the data message, while in other embodiments, thefirewall service engine performs its firewall action on the datamessage.

In some embodiments, other MFN components direct the firewall serviceengine to perform its operation. For instance, at an ingress node, theVPN gateway (e.g., 225 or 230) in some embodiments directs itsassociated firewall service engine to perform its operation, in order todetermine whether the data message should be passed to the ingressnode's CFE. Also, at the egress node, the CFE in some embodiments passesthe data message to its associated firewall service engine, which if itdecides to allow the data message through, then passes the data messagethrough an external network (e.g., the Internet) to its destination, orpasses the data message to its associated NAT engine 215 to perform itsNAT operation before passing the data message to its destination throughan external network.

The virtual network providers of some embodiments allow the tenant's WANsecurity gateway that is defined in the public clouds to implement othersecurity services in addition to, or instead of, firewall services. Forinstance, a tenant's distributed WAN security gateway (which in someembodiments is distributed over each public cloud datacenter that isspanned by the tenant's virtual network) not only includes firewallservice engines, but also includes intrusion detection engines andintrusion prevention engines. In some embodiments, the intrusiondetection engines and intrusion prevention engines are incorporatedarchitecturally in the MFN 150 to occupy similar position to thefirewall service engine 210.

Each of these engines in some embodiments includes one or more storagesthat store intrusion detection/prevention policies distributed by thecentral controller cluster 160. In some embodiments, these policiesconfigure the engines to detect/prevent unauthorized intrusions into thetenant's virtual network (that is deployed over several public clouddatacenters), and to take actions in response to detected intrusionevents (e.g., generating logs, sending out notifications, shutting downservices or machines, etc.). Like firewall rules, the intrusiondetection/prevention policies can be enforced at various differentmanaged forwarding nodes (e.g., ingress MFNs, intermediate MFNs, and/oregress MFNs of the data message flows) over which the virtual network isdefined.

As mentioned above, the virtual network provider deploys each tenant'svirtual WAN by deploying at least one MFN in each public cloud spannedby the virtual WAN, and configuring the deployed MFNs to define routesbetween the MFNs that allow the tenant's message flows to enter and exitthe virtual WAN. Also, as mentioned above, each MFN can be shared bydifferent tenants in some embodiments, while in other embodiments eachMFN is deployed for just one particular tenant.

In some embodiments, each tenant's virtual WAN is a secure virtual WANthat is established by connecting the MFNs used by that WAN throughoverlay tunnels. This overlay tunnel approach in some embodimentsencapsulates each tenant's data message flows with a tunnel header thatis unique to each tenant, e.g., contains a tenant identifier thatuniquely identifies the tenant. For a tenant, the virtual networkprovider's CFEs in some embodiments use one tunnel header to identifyingress/egress forwarding elements for entering/exiting the tenant'svirtual WAN, and another tunnel header to traverse interveningforwarding elements of the virtual network. The virtual WAN's CFEs usedifferent overlay encapsulation mechanisms in other embodiments.

To deploy a virtual WAN for a tenant over one or more public clouds, theVNP's controller cluster (1) identifies possible edge MFNs (that canserve as ingress or egress MFNs for different data message flows) forthe tenant based on locations of the tenant's corporate compute nodes(e.g., branch offices, datacenters, mobile users, and SaaS providers),and (2) identifies routes between all possible edge MFNs. Once theseroutes are identified they are propagated to the forwarding tables ofthe CFEs (e.g., propagated using OpenFlow to different OVS-based virtualnetwork routers). Specifically, to identify optimal routes through atenant's virtual WAN, the MFNs associated with this WAN generatemeasurement values that quantify the quality of the network connectionbetween them and their neighboring MFNs, and regularly provide theirmeasurements to the VNP's controller cluster.

As mentioned above, the controller cluster then aggregates themeasurements from the different MFNs, generates routing graphs based onthese measurements, defines routes through a tenant's virtual WAN, andthen distributes these routes to the forwarding elements of the CFEs ofthe MFNs. To dynamically update the defined routes for a tenant'svirtual WAN, the MFNs associated with this WAN periodically generatetheir measurements and provide these measurements to the controllercluster, which then periodically repeats its measurement aggregation,route-graph generation, route identification, and route distributionbased on the updated measurements that it receives.

In defining the routes through a tenant's virtual WAN, the VNP'scontroller cluster optimizes the routes for the desired end-to-endperformance, reliability and security, while trying to minimize therouting of tenant's message flows through the Internet. The controllercluster also configures the MFN components to optimize the layer 4processing of the data message flows passing through the network (e.g.,to optimize the end-to-end rate of TCP connections by splitting the ratecontrol mechanisms across the connection path).

With the proliferation of public clouds, it is often very easy to find amajor public cloud datacenter close to each branch office of acorporation. Similarly, SaaS vendors are increasingly hosting theirapplications within public clouds, or are similarly located at thevicinity of some public cloud datacenter. Consequently, the virtualcorporate WANs 1415 securely use the public clouds 1405 as corporatenetwork infrastructure that have presence in the vicinity of thecorporate compute nodes (e.g., branch offices, datacenters, remotedevices, and SaaS providers).

Corporate WANs require bandwidth guarantees in order to provide businesscritical application at an acceptable performance at all times. Suchapplications maybe interactive data applications, e.g. ERP, financial orprocurement, deadline-oriented application (e.g., industrial or IoTcontrol), real time application (e.g., VoIP or video conferencing).Consequently, traditional WAN infrastructure (e.g., Frame Relay or MPLS)provides such guarantees.

A main obstacle in providing bandwidth guarantee in a multi-tenantnetwork (e.g., a shared virtual network over which multiple virtualnetworks are defined) is the need to reserve bandwidth over one or morepath for a certain customer. In some embodiments, the VNP offers QoSservices and provides an Ingress Committed Rate (ICR) guarantee and anEgress Committed Rate (ECR) guarantee. ICR refers to the traffic ratecoming into the virtual network, while ECR refers to the traffic rateexiting the virtual network to the tenant site.

As long as traffic does not exceed ICR and ECR limits, the virtualnetwork in some embodiments provides bandwidth and delay guarantees. Forexample, as long as HTTP ingress or egress traffic do not exceed 1 Mbps,the bandwidth and low delay are guaranteed. This is the point-to-cloudmodel because, for QoS purposes, the VNP need not keep track of trafficdestinations, as long as its destinations are within the ICR/ECR bounds.This model is sometimes called the hose model.

For the more stringent applications, where a customer desires apoint-to-point guarantee, a virtual data pipe needs to be constructed todeliver the highly critical traffic. For example, an enterprise may wanttwo hub sites or datacenters connected with high service level agreementguarantees. To that end, VNP routing automatically chooses a routingpath that satisfies the bandwidth constraint for each customer. This isreferred to as the point-to-point model or the pipe model.

The main advantage of VNP in providing guaranteed bandwidth to end usersis the ability to adjust the VNP infrastructure according to thechanging bandwidth demands. Most public clouds provide minimum bandwidthguarantees between each two instances located at different regions ofthe same cloud. If the current network does not have enough unusedcapacity to provide the guaranteed bandwidth for a new request, the VNPadds new resources to its facilities. For example, the VNP can add newCFEs in high-demand regions.

One challenge is to optimize the performance and the cost of this newdimension in planning routes and scaling up and down the infrastructure.To facilitate the algorithms and bandwidth accounting, some embodimentsassume that end-to-end bandwidth reservations are not split. In otherways, if a certain bandwidth (e.g., 10 Mbps) is reserved between branchA and branch B of a certain tenant, the bandwidth is allocated over asingle path that starts from an ingress CFE to which branch A connects,and then traverses a set of zero or more intermediate CFEs to reach theegress CFE that is connected to branch B. Some embodiments also assumethat the bandwidth guaranteed path only traverse a single public cloud.

In order to account for the various bandwidth reservation that intersectover the network topology, the VNP in some embodiments defines therouting over a reserved bandwidth path statically, so that data messageflows always traverse through the same routes that were reserved for thebandwidth requirements. In some embodiments, each route is identifiedwith a single tag that each CFE traversed by the route matches to asingle outgoing interface associated with this route. Specifically, eachCFE matches a single outgoing interface to each data message that hasthis tag in its header and arrives from a specific incoming interface.

In some embodiments, the controller cluster maintains a network graphthat is formed by several interconnected nodes. Each node n in the graphhas the allocated total guaranteed bandwidth (TBW_(n)) associated withthis node and the amount of bandwidth already reserved (allocated to acertain reserved path) by this node (RBW_(n)). In addition, for eachnode, the graph includes the cost in cents per gigabyte (C_(ij)) and thedelay in milliseconds (D_(ij)) associated with sending traffic betweenthis node and all other nodes in the graph. The weight associated withsending traffic between node i and node j is W_(ij)=a*C_(ij)+D_(ij),where a is a system parameter that is typically between 1 and 10.

When a request for a bandwidth reservation of value BW between branchesA and B is accepted, the controller cluster first maps the request tospecific ingress and egress routers n and m, which are bound to branchesA and B respectively. The controller cluster then executes a routingprocess that conducts two lowest-cost (e.g., shortest path) computationsbetween n and m. The first is a lowest-cost (e.g., shortest path) routebetween n and m irrespective of the available bandwidth along thecomputed route. The total weight of this route is computed as Wi.

The second lowest-cost (e.g., shortest path) computation initiallymodifies the graph by eliminating all nodes i where BW>TBW_(i)−RBW_(i).The modified graph is termed the trimmed graph. The controller clusterthen performs a second lowest-cost (e.g., shortest path) routecomputation over the trimmed graph. If the weight of the second route isno more than K percent (K is typically 10%-30%) higher than the firstroute, the second route is selected as the preferred path. On the otherhand, when this requirement is not met, the controller cluster will addto the first path the node i with the smallest value of TBW_(i)−RBW_(i),and then repeats the two lowest-cost (e.g., shortest path) computations.The controller cluster will continue adding more routers until thecondition is met. At that point, the reserved bandwidth BW is added toall RBW_(i) where i is a router on the selected route.

For the special case of a request for additional bandwidth for a routethat already has reserved bandwidth, the controller cluster will firstdelete the current bandwidth reservation between nodes A and B and willcalculate the path for the total bandwidth request between these nodes.To do this, the information held for each node in some embodiments alsoincludes the bandwidth reserved for each tag, or each source anddestination branches, and not only the overall bandwidth reserved. Afterbandwidth reservations are added to the network, some embodiments do notrevisit the routes so long as there are no major changes in measurednetwork delays or costs through the virtual network. However, when themeasurements and/or costs change, these embodiments repeat the bandwidthreservation and route computation processes.

FIG. 15 conceptually illustrates a process 1500 performed by thecontroller cluster 160 of the virtual network provider to deploy andmanage a virtual WAN for a particular tenant. In some embodiments, theprocess 1500 is performed by several different controller programsexecuting on the controller cluster 160. The operations of this processdo not necessarily have to follow the sequence illustrated in FIG. 15,as these operations can be performed by the different programs inparallel or in a different sequence. Accordingly, these operations areillustrated in this figure only to describe one exemplary sequence ofoperations performed by the controller cluster.

As shown, the controller cluster initially deploys (at 1505) severalMFNs in several public cloud datacenters of several different publiccloud providers (e.g., Amazon AWS, Google GCP, etc.). The controllercluster in some embodiments configures (at 1505) these deployed MFNs forone or more other tenants that are different than the particular tenantfor which the process 1500 is illustrated.

At 1510, the controller cluster receives from the particular tenant dataabout external machine attributes and locations of the particulartenant. In some embodiments, this data includes the private subnets usedby the particular tenant as well as identifiers for one or more tenantoffices and datacenters at which the particular tenant has externalmachines. In some embodiments, the controller cluster can receive thetenant data through APIs or through a user interface that the controllercluster provides.

Next, at 1515, the controller cluster generates a routing graph for theparticular tenant from the measurements collected by the measurementagents 205 of the MFNs 150 that are candidate MFNs to use forestablishing the virtual network for the particular tenant. As mentionedabove, the routing graph has nodes that represent the MFNs, and linksbetween the nodes that represent the network connections between theMFNs. The links have associated weights, which are cost values thatquantify the quality and/or cost of using the network connectionsrepresented by the links. As mentioned above, the controller clusterfirst generates a measurement graph from the collected measurements, andthen generates the routing graph by removing links from the measurementgraph that are not optimal (e.g., that have large delays or drop rates).

After constructing the routing graph, the controller cluster performs(at 1520) path searches to identify possible routes between differentpairs of candidate ingress and egress nodes (i.e., MFNs) that thetenant's external machines can use to send data messages into thevirtual network (deployed by the MFNs) and to receive data messages fromthe virtual network. In some embodiments, the controller cluster usesknown path search algorithms to identify different paths between eachcandidate ingress/egress pair of nodes. Each path for such a pair usesone or more links that when concatenated traverse from the ingress nodeto the egress node through zero or more intermediate nodes.

In some embodiments, the cost between any two MFNs comprises a weightedsum of estimated latency and financial costs for a connection linkbetween the two MFNs. The latency and financial costs include in someembodiments one or more of the following: (1) link delay measurements,(2) estimated message processing latency, (3) cloud charges for outgoingtraffic from a particular datacenter either to another datacenter of thesame public cloud provider, or to exit the public cloud (PC) provider'scloud (e.g., to another public cloud datacenter of another public cloudprovider or to the Internet), and (4) estimated message processing costsassociated with the MFNs executing on host computers in the publicclouds.

Some embodiments assess a penalty for connection links between two MFNsthat traverse through the public Internet, in order to minimize suchtraversal whenever possible. Some embodiments also incentivize the useof private network connections between two datacenters (e.g., byreducing the connection link cost) in order to bias the route generationtowards using such connections. Using the computed costs of thesepair-wise links, the controller cluster can compute the cost of eachrouting path that uses one or more of these pair-wise links byaggregating the costs of the individual pair-wise links that are used bythe routing path.

The controller cluster then selects (at 1520) one or up to N identifiedpaths (where N is an integer larger than 1) based on the computed costs(e.g., the lowest aggregate cost) of the identified candidate pathsbetween each candidate ingress/egress pair of nodes. In someembodiments, the computed costs for each path are based on the weightcost of each link used by the path (e.g., is a sum of each link'sassociated weight value), as mentioned above. The controller cluster canselect more than one path between a pair of ingress/egress nodes whenmore than one route is needed between two MFNs to allow the ingress MFNor an intermediate MFN to perform a multi-path operation.

After selecting (at 1520) one or N paths for each candidate pair ofingress/egress nodes, the controller cluster defines one or N routesbased on the selected paths, and then generates route tables or routetable portions for the MFNs that implement the particular tenant'svirtual network. The generated route records identify edge MFNs to reachdifferent subnets of the particular tenant, and identify next hop MFNsfor traversing routes from ingress MFNs to egress MFNs.

At 1525, the controller cluster distributes route records to the MFNs inorder to configure the forwarding elements 235 of these MFNs toimplement the virtual network for the particular tenant. In someembodiments, the controller cluster communicates with the forwardingelements to pass the route records by using communication protocols thatare presently used in a software defined multi-tenant datacenter toconfigure software routers executing on host computers to implement alogical network that spans the host computers.

Once the MFNs have been configured and the virtual network isoperational for the particular tenant, the edge MFNs receive datamessages from tenant's external machines (i.e., machines outside of thevirtual network) and forward these data messages to edge MFNs in thevirtual network, which in turn forward the data messages to otherexternal machines of the tenant. While performing such forwardingoperations, the ingress, intermediate and egress MFNs collect statisticsregarding their forwarding operations. Also, in some embodiments, one ormore modules on each MFN in some embodiments collect other statisticsregarding network or compute consumption in the public clouddatacenters. In some embodiments, the public cloud providers collectsuch consumption data and pass the collected data to the virtual networkprovider.

When approaching a billing cycle, the controller cluster collects (e.g.,at 1530) statistics collected by the MFNs, and/or the network/computeconsumption data collected by the MFNs or provided by the public cloudproviders. Based on the collected statistics, and/or provided thenetwork/compute consumption data, the controller cluster generates (at1530) billing reports and sends the billing reports to the particulartenant.

As mentioned above, the amount billed in the billing report accounts forstatistics and network/consumption data that the controller clusterreceives (e.g., at 1530). Also, in some embodiments, the bill accountsfor the cost that the virtual network provider incurred to operate theMFNs (that implement the virtual network for the particular tenant) plusa rate of return (e.g., a 10% increase). This billing scheme isconvenient for the particular tenant because the particular tenant doesnot have to deal with bills from multiple different public cloudproviders over which the tenant's virtual network is deployed. The VNP'sincurred cost in some embodiments includes the cost charged to the VNPby the public cloud providers. At 1530, the controller cluster alsocharges a credit card or electronically withdraws funds from a bankaccount for the charges reflected in the billing report.

At 1535, the controller cluster determines whether it has received newmeasurements from the measurement agents 205. If not, the processtransitions to 1545, which will be described below. On the other hand,when the controller cluster determines that it has received newmeasurements from the measurement agents, it determines (at 1540)whether it needs to re-examine its routing graph for the particulartenant based on the new measurements. Absent an MFN failure, thecontroller cluster in some embodiments at most updates its routing graphfor each tenant once during a particular time period (e.g., once every24 hours or every week) based on received, updated measurements.

When the controller cluster determines (at 1540) that it needs tore-examine the routing graph based on new measurements that it hasreceived, the process generates (at 1545) a new measurement graph basedon the newly received measurements. In some embodiments, the controllercluster uses a weighted sum to blend each new measurement with the priormeasurements in order to ensure that the measurement values associatedwith the links of the measurement graph do not fluctuate dramaticallyeach time a new measurement set is received.

At 1545, the controller cluster also determines whether it needs toadjust the routing graph based on the adjusted measurement graph (e.g.,whether it needs to adjust weight values for the routing-graph links, oradd or remove links in the routing graph because of adjusted measurementvalues associated with the links). If so, the controller cluster (at1545) adjusts the routing graph, performs path search operations (suchas operations 1520) to identify routes between ingress/egress nodepairs, generates route records based on the identified routes, anddistributes route records to the MFNs. From 1545, the processtransitions to 1550.

The process also transitions to 1550 when the controller clusterdetermines (at 1540) that it does not need to re-examine the routinggraph. At 1550, the controller cluster determines whether it has tocollect statistics regarding data messages processed and network/computeresources consumed. If not, the process returns to 1535 to determinewhether it has received new measurements from the MFN measurementagents. Otherwise, the process returns to 1530 to collect statistics andnetwork/compute consumption data, and to generate and send billingreports. In some embodiments, the controller cluster repeatedly performsthe operations of the process 1500 until the particular tenant no longerneeds a virtual network that is deployed across the public clouddatacenters.

In some embodiments, the controller cluster not only deploys virtualnetworks for tenants in the public cloud datacenters, but also assiststhe tenants in deploying and configuring compute node machines andservice machines in the public cloud datacenters. The deployed servicemachines can be machines separate from the service machines of the MFNs.In some embodiments, the controller cluster billing report to theparticular tenant also accounts for compute resources consumed by thedeployed compute and service machines. Again, having one bill from onevirtual network provider for network and compute resources consumed inmultiple public cloud datacenters of multiple public cloud providers ismore preferable for the tenant than receiving multiple bills frommultiple public cloud providers.

Other embodiments use other deployment models to deploy dedicated orshared MFNs to implement dedicated or shared virtual networks over thenetwork and compute infrastructure of two or more public cloudproviders. For instance, in some embodiments, the virtual networkprovider allows one or more cloud service resellers to deploy dedicatedor shared MFNs for one or more of their customers. In some embodiments,a dedicated virtual network is one that is established with dedicatedMFNs that are deployed for one entity. On the other hand, in some ofthese embodiments, a shared virtual network is one that is establishedwith shared MFNs that are deployed for use by multiple entities.Multiple segregated virtual networks for multiple entities can bedefined over a shared virtual network in some embodiments.

FIG. 16 illustrates one such deployment model. As shown, this deploymentmodel uses three levels of SaaS providers 1605-1615 that provide threesets of SaaS services. The first SaaS layer 1605 is provided by one ormore public cloud providers 1620 that provide compute and networkinfrastructure (e.g., compute elements (such as host computers, VMsand/or containers) and network elements (hardware or software switches,routers, middlebox elements, etc.) that connect the compute elements) inmultiple different public clouds 1650. The second SaaS layer 1610 isprovided by the VNP 1625, which provides the tools for deploying virtualnetworks across multiple public clouds 1650 for a cloud reseller. Thecloud reseller 1630 provides the third SaaS layer 1615 to its customers1640, which use the cloud resellers tools to define compute elements andnetwork infrastructure (e.g., virtual network) to deploy across one ormore public clouds 1650.

The cloud reseller 1630 in some embodiments has its own customer accountwith each of the public cloud providers. In some embodiments, the cloudreseller establishes its customer account with each public cloudprovider directly (as identified by dashed arc 1652) and then providessecurity credentials for this customer account to the VNP provider 1625.In other embodiments, the cloud reseller establishes its public cloudcustomer account through the VNP provider's machines 1625. To direct theVNP to deploy a virtual network for one of its tenants over public cloudinfrastructure, the cloud reseller machines 1630 initially provide thecloud reseller's VNP security credentials (e.g., username, password,certificate, etc.) to the VNP machines 1625 in order to authenticate thecloud reseller.

As shown, the cloud reseller also provides the desired networkconfiguration to the VNP 1625. This configuration described theattributes of the virtual network that needs to be deployed for atenant. In some embodiments, this configuration data also includes atenant identifier that the cloud reseller 1630 uses for its customer forwhich it directs the VNP 1625 to deploy a virtual network. This tenantidentifier in some embodiments is obtained from the VNP. In otherembodiments, the configuration data includes a tenant identifierprovided by the VNP, which the cloud reseller maps to its own tenantidentifier for its tenant.

In some embodiments, the VNP machines 1625 provide the cloud reseller'ssecurity credentials to the public cloud machines 1620 when deployingvirtual networks (e.g., when deploying MFNs, configuring MFN CFEs withproper routing records, etc.) for the reseller's customers. Thesesecurity credentials (e.g., user name, password, security certificate,etc.) allow the VNP machines to authenticate themselves as machinesoperating at the behest of the cloud reseller (e.g., to logon onto thepublic cloud machines 1620 as if the cloud reseller machines 1630 arelogging on to the public cloud machines 1620).

To deploy the virtual network for each tenant of the cloud reseller, theVNP machines also configure in some embodiments the components that formthe virtual network (e.g., the MFN, MFN gateways/service boxes/CFEs,etc.) with the tenant identifier for the cloud reseller's tenant. Insome embodiments, these configured components then associate thestatistics that they collect (e.g., the routing statistics anddeployment machine compute statistics) with the tenant identifiers, sothat the cloud reseller's customers can be appropriately charged basedon these statistics, as further described below.

In some of the embodiments in which the cloud reseller provides thetenant identifiers for its customers to the VNP machines 1625, the VNPmachines 1625 map each tenant identifier provided by the cloud reseller1630 to a unique tenant identifier of the VNP, and translate the VNPtenant identifiers for the collected statistics to the cloud reseller'stenant identifiers before providing the statistics to the cloudreseller. In other embodiments, the cloud reseller machines 1630 and/orVNP provider machines 1625 use tenant identifiers provided by the publiccloud provider machines 1620.

The VNP machines 1625 uses the tenant network configuration dataprovided by the cloud reseller 1630 to deploy and/or configure thevirtual network components (e.g., the MFNs, etc.) of the virtual networkfor each customer of the cloud reseller. In some embodiments, multipledifferent customers share the same virtual network and/or virtualnetwork components. Conjunctively, or alternatively, the VNP machines1625 in some embodiments can define a dedicated virtual network for anyindividual customer of the cloud reseller. As shown, the VNP machines1625 deploy and configure the virtual networks and/or virtual networkcomponents through the public cloud provider machines 1620. In someembodiments, reseller, VNP, and public cloud machines 1620-1630communicate with each other through their respective API interfaces andintervening network fabric (e.g., the Internet).

The VNP machines 1625 collect per tenant statistics from the deploymentvirtual network components (e.g., routing statistics from the CFEs andgateways, etc.), aggregate the statistics, and forward the aggregatedstatistics to the cloud reseller. In the embodiments in which the VNPmachines map the reseller customer identifiers to VNP tenantidentifiers, the VNP machines translate the tenant identifiers to thecustomer identifiers before supplying the aggregated statistics with thetranslated customer identifiers. Also, in the embodiments in which theVNP machines map the public cloud identifiers to VNP tenant identifiers,the VNP machines translate the public cloud identifiers to the VNPidentifiers.

As shown, the VNP machines 1625 periodically send billing reports to thecloud reseller 1630 to collect fees for services performed by the VNP.In some embodiments, these billing reports charge the cloud reseller forthe VNP's service fees for deploying virtual networks for the cloudreseller over two or more public clouds. These deployment chargesinclude fees for performing ancillary operations in support of suchdeployments, such as measurement operations that produce measurementsthat quantify the quality and/or cost of links between MFNs in thepublic clouds and between external machine locations of the tenants.

Also, the VNP machines 1625 in some embodiments receive billing datafrom one or more public cloud providers for the cloud reseller. Thisbilling data is associated with the cloud reseller's customercredentials (e.g., PC provider customer number for the cloud reseller)in some embodiments. The VNP machines 1625 in these embodiments passalong the billing data to the cloud reseller (e.g., with a markupadjustment or without a markup adjustment). In other embodiments, thepublic cloud providers send billing reports directly to the cloudreseller machines 1630, as shown by dashed lines 1652.

The cloud reseller machines 1630 in some embodiments uses the usagestatistics provided by the VNP machines 1625 to charge its customers forthe virtual networks. In some embodiments, the VNP machines 1625 notonly deploy network infrastructure but also deploy computeinfrastructure for the cloud reseller 1630. In some of theseembodiments, the usage statistics reflects the used compute resourcesand the cloud reseller machines 1630 use these statistics to charge thereseller's customers. In some embodiments, the cloud reseller does notuse the collected statistics to charge its customers, but rather chargesits customers based on the compute and/or network configuration that thecustomer requested for deployment.

To further illustrate the differences between the three-layer SaaSdeployment model of FIG. 16, FIG. 17 illustrates a similar diagram forthe two-layer SaaS deployment model that was previously described above.This two-layer model of FIG. 17 does not have any cloud reseller 1615.Rather, in this two-layer model, the VNP's customers are entities thathave the VNP deploy a dedicated virtual network just for them, or use ashared virtual network that the VNP has deployed for several customers.As described above, different customers can securely share thecomponents the define the shared virtual network over one or more publicclouds, as each customers network traffic is securely segregated fromthe network traffic of other customers. In some embodiments, multiplevirtual networks are defined for multiple customers over the sharedvirtual network by using tenant identifiers to segregate the traffic ofthe different customers, as described above.

In the two-layer model of FIG. 17, the first SaaS layer 1605 is providedby one or more public cloud providers 1620 that provide compute andnetwork infrastructure in multiple different public clouds 1650, whilethe second SaaS layer 1610 is provided by the VNP 1625, which providesthe tools for deploying virtual networks across multiple public clouds1650 for several of its customers. As shown, the VNP machines 1625provide to the public cloud providers the VNP security credentials forthe public clouds.

The VNP machines receive for each customer (associated with a tenantidentifier) tenant network configuration data, and based on this data,deploy and/or configure the virtual network components (e.g., the MFNs,etc.) of the virtual network for each of its customers. As shown, theVNP machines 1625 deploy and configure the virtual networks and/orvirtual network components through the public cloud provider machines1620. In some embodiments, VNP and and public cloud machines 1620 and1625 communicate with each other through their respective API interfacesand intervening network fabric (e.g., the Internet).

As further shown, the VNP machines 1625 collect per tenant statisticsfrom the deployment virtual network components (e.g., routing statisticsfrom the CFEs and gateways, etc.), and aggregate the collectedstatistics. The VNP machines 1625 periodically send billing reports toeach of the VNP customers to collect fees for services performed by theVNP. As mentioned above, these fees in some embodiments include the feesthat the public cloud providers charged the VNP for the resources (e.g.,compute and/or network resources) consumed by the customer's virtualnetwork, plus a certain markup percentage. The VNP machines identify theamount of resources by each customer's virtual network based on thestatistics that these machines collect and aggregate for the customer'sassociated identifier. In other embodiments, the VNP machines passthrough to each customer each public cloud provider's charge for theresources consumed by the customer's virtual network, plus a charge foreach customer's use of VNP resources.

Some embodiments connect a multi-machine compute node (e.g., a branchoffice or datacenter) of a tenant to the tenant's public cloud virtualnetwork through multiple connection links to multiple public clouds(e.g., multiple public cloud datacenters) of one or more public cloudproviders. Having multiple links between the multi-machine compute node(MMCN) and the public cloud virtual network allows the virtual networkto be highly available, as it can withstand the failure of one or moreconnection links. In some embodiments, one link is a primary link whileeach other link is a failover link. This approach also allows the bestroute to be established from each MMCN to each other MMCN or SaaSprovider datacenter by selecting the best ingress node for entering thevirtual network from the MMCN for the best routing path through thevirtual network to the egress node for exiting the virtual network toanother MMCN or SaaS provider datacenter.

The discussion below uses the term multi-homed MMCN to refer to amulti-machine compute node of a tenant that connects to the tenant'spublic cloud virtual network through multiple connection links tomultiple public clouds of one or more public cloud providers. Thediscussion below also uses the term multi-homed SaaS datacenter to referto a SaaS datacenter to which a virtual network associates multiple MFNsin one or more public clouds (e.g., multiple public cloud datacenters)of one or more public cloud providers. These MFNs in some embodimentsare candidate egress nodes for routes that traverse through the virtualnetwork to reach a SaaS provider. The use of two or more egress nodes toconnect the virtual network of a SaaS datacenter is also advantageous inthat it enables link failover support and it allows for the use ofoptimal routes between different pairs of external computer node (e.g.,remote computer or MMCN) and SaaS provider datacenter.

In some embodiments, a SaaS datacenter does not need to initiate routesto multiple MFNs of the virtual network in multiple public clouddatacenters, even when the virtual network controllers 160 associatemultiple MFNs with the SaaS datacenter. On the other hand, multi-homedMMCNs in some embodiments need to actively initiate routes throughdifferent links to the virtual network. To do this, providing a fallbackcapability is facilitated in a multi-homed MMCN by having an appropriaterouter with failover capabilities (e.g., by using a Cisco 2800 series).

For optimal routing, the multi-homed MMCN includes in some embodimentsone or more computers or appliances that execute measurement processesto measure the performance (delay, loss etc.) between the MMCN and thedifferent public cloud datacenters to which the MMCN can connect. Inaddition, the MMCN in some embodiments performs its overall routingoperations based on routes that are defined by the centralizedcontroller cluster (e.g., controller cluster 160) that defines thevirtual network for an entity (e.g., for a tenant). To accomplish this,the multi-homed MMCN is equipped with SD-WAN capabilities (such asVelocloud and Viptela appliances) that operate as a part of thecentralized control plane for deploying the virtual network. Asmentioned above, the centralized control plane is implemented by acluster of two or more controllers in some embodiments.

FIG. 18 illustrates a process 1800 used by the central controllercluster of some embodiments to define routes for a particularmulti-homed MMCN. This process uses a specialized router in theparticular multi-homed MMCN to use the defined routes to perform routingoperations that forward data messages from the MMCN to the virtualnetwork through the multiple connection links. The specialized router isa software router or a cluster of software routers in some embodiment,while it is a routing appliance (e.g., SD-WAN appliance) in otherembodiments. The specialized router or router cluster in the MMCN isreferred to as the edge node of the MMCN in the discussion below. Insome embodiments, the central controller cluster remotely controls theedge nodes of the MMCNs through the Internet.

As shown, the central controller cluster initially identifies (at 1805)a subset of N MFNs (e.g., 10 to 12) from N different cloud regions thatare the closest to the particular MMCN edge node according to a DNSserver service and the edge node IP address. The N MFNs in someembodiments have at least one candidate MFN from each cloud providerwithin a certain distance of the particular MMCN. Also, in someembodiments, each of the N MFNs includes a gateway (e.g., a branchgateway 225) to establish a secure connection link with the particularMMCN edge node.

Also, as mentioned above, the DNS server service in some embodiments isa service machine, or a cluster of several service machines, thatoperates in one or more public clouds and that provides DNS informationto DNS servers of the MMCNs of an entity. In some of these embodiments,the DNS servers are operated by the virtual network provider or byanother entity. In other embodiments, the DNS server service is a geo-IPservice (e.g., of a third party) that resides outside of the publicclouds that implement the virtual network and that can identify edgenodes in the public clouds that are near the particular MMCN for whichthe process 1800 is performed.

Next, at 1810, the controller cluster downloads the identified list to Nnodes to the particular MMCN's edge node so that the MMCN's edge nodecan take measurements that quantify the quality of connections to eachof the N MFNs in the list. In some embodiments, each MMCN edge node hasa measurement agent (e.g., a process executing on one of the MMCNcomputers) that generates such measurements. This measurement agentgenerates measurement values differently in different embodiments. Insome embodiments, the measurement agent sends pinging messages (e.g.,UDP echo messages) periodically (e.g., once every second, every Nseconds, every minute, every M minutes, etc.) to each of the measurementagents of the N MFNs in the list. Based on the speed of the replymessages that it receives, the MMCN measurement agent computes andupdates measurement metric values, such as network-connection throughputspeed, delay, loss, and link reliability. In some embodiments, multipleMFNs share one measurement agent (e.g., in the same datacenter or nearbydatacenter of the public cloud provider hosting the MFNs).

In some embodiments, the particular MMCN's measurement agentperiodically performs these measurements, and periodically sends the newmeasurements to the controller cluster so that the controller clustercan update its weight computations and route generations, as furtherdescribed below by reference to 1820-1835. Also, whenever new MFNs areadded in newly added or previously used public cloud datacenters, thecontroller cluster in some embodiments generates update lists of Ncandidate MFNs.

At 1815, the controller cluster receives the measurements taken by theparticular MMCN's edge node. Based on these measurements, thecentralized controller cluster computes (at 1820) a link weight for eachconnection link that connects the particular MMCN's edge node to each ofthe N MFNs. For instance, in some embodiments, the central controllercomputes each link's weight by using an exponential filter on the delaymeasurements and using the loss parameter as weight multiplier (e.g.,doubling the weight for each 1% of loss).

Based on the computed weights, the central controller then identifies(at 1825) a subset of the M (e.g., 5 or 6) MFNs as the “home” nodes towhich the edge node will be connected. In some embodiments, the M nodesare the nodes with the lowest weight values. In other embodiments, the Mnodes are the nodes with the lowest weight values, but at least onerepresentative MFN in each cloud provider is included in the M nodes.The list of M nodes may change with time and MFNs can be dropped andadded to the list as new MFNs are added and/or as new measurements arereceived from the particular MMCN edge node. The controller cluster insome embodiments uses a “hysteresis” process to avoid frequent changesin the list of M MFNs. The hysteresis process in some embodiments usesprevious state (i.e., previous members) of the MFN list to reduce therate of adding/removing members to/from the MFN list. Also, in someembodiments, the controller cluster will not drop an MFN from the listunless another MFN has a 10% smaller average weight for a window (e.g.,a time period) of measurements.

As mentioned above, the particular MMCN edge node in some embodimentsmaintains a secure connection (e.g., an IPsec connection) to the virtualnetwork gateway of each of the M MFNs. In some embodiments, thecontroller cluster directs (at 1825) the particular MMCN edge node toestablish secure connections with each of the M MFNs. At 1830, thecontroller cluster uses the computed weights of the selected M MFNs toidentify optimal routes and failover routes for connecting theparticular MMCN edge node with each other possible nodes for datamessage flows to traverse between the particular MMCN edge node andother MMCN edge nodes or SaaS provider datacenters through the virtualnetwork. To generate such routes, the controller cluster in someembodiments uses shortest path route-identification processes, asdescribed above.

In some embodiments, the controller cluster repeats its routeidentification process periodically, or whenever the computed weightvalues change (e.g., based on new measurements or addition/deletion ofMFNs in the list of M MFNs). In some embodiments, the controller clusterperforms the route identification operation (at 1830) for the particularMMCN edge node along with the route identification operation for othermulti-homed MMCNs and/or multi-homed SaaS providers together, asmultiple connection links to other MMCNs and to the SaaS providers wouldbe relevant in identifying optimal routes to and from the particularMMCN. These computed routes also account for routes to/from virtualnetwork MFNs that are candidates for connecting to remote devices (e.g.,remote laptops, desktops, or mobile devices, such as smartphones,tablets, etc.).

After identifying these routes, the controller cluster supplies (at1835) forwarding records for one or more routes to the particular MMCNedge node and the MFNs. For instance, in some embodiments, thecontroller cluster provides forwarding records (e.g., routing recordsthat specify the next hop, routing records that specify the virtualnetwork egress node, etc.) to particular MMCN edge node and to the MFNCFEs. By using these forwarding records to perform their routingoperations, the particular MMCN edge node and MFN CFEs implement theoptimal and failover routes defined (at 1830) by the controller cluster.In some embodiments, the controller cluster supplies new routing recordsto the particular MMCN edge node and the MFN CFEs whenever it identifiesnew optimal or failover routes.

In this manner, the process 1800 of FIG. 18 bases its routingcomputations on computed weight values that express the quality of theconnection between the particular MMCN and each of its severalconnections to the virtual network. Under this approach, a differentvirtual-network ingress/egress node pair can be selected for theparticular MMCN edge node and different MMCNs, different SaaS nodesand/or different remote devices. Because of this, the controller clusterin some embodiments performs the route identification operation (i.e.,operation 1830) for one or more multi-homed MMCNs and/or multi-homedSaaS providers together, as mentioned above.

FIG. 19 presents an example of two branch nodes EN1 and EN2 of two MMCNs1905 and 1910 and a SaaS datacenter 1915. Each of the branch nodesconnects to the virtual network 1920 through virtual-network MFNs1925-1935 that are defined in three public cloud datacenters of two orthree public cloud providers. The SaaS datacenter 1915, on the otherhand, can be accesses through virtual-network MFNs 1940 and 1945. Theweight measured between the relevant branch nodes EN1 and EN2, MFNs1925-2545 and SaaS datacenter 1915 are depicted on the links betweenthese nodes. In this example, it is assumed that other weights, likebetween nodes 1925 and 1935 are much higher (e.g. 10), so that noshortest path routing algorithm will use them in the best cost path.

As can be seen from this example, the best path from EN1 to the SaaSdatacenter traverses nodes 1925 and 1940 as this path has a weight sumof 14, which is smaller than other weight costs of other paths. Forinstance, going through node 1930 will incur a smaller weight in thefirst hop but will result in a total minimal weight of 15. The optimalroute from branch node EN2 will be through router 1935 and 1945 with atotal weight of 15. Consequently, the two branches will use twodifferent routes to reach the SaaS datacenter. To communicate betweenEN1 and EN2, the best route will be through MFN 1930 with a total weightof 13.

As mentioned above, some embodiments associate two or morevirtual-network MFNs with each SaaS provider's datacenter. SaaS is asoftware distribution model in which a third-party provider hostsapplications and makes them available to customers over the Internet.SaaS removes the need for organizations to install and run applicationson their own computers or in their own datacenters. This eliminates theexpense of hardware acquisition, provisioning and maintenance, as wellas software licensing, installation and support. Also, rather thanpurchasing software to install, or additional hardware to support it,customers subscribe to a SaaS offering. Generally, they pay for thisservice on a monthly basis using a pay-as-you-go model. Transitioningcosts to a recurring operating expense allows many businesses toexercise better and more predictable budgeting. Users can also terminateSaaS offerings at any time to stop those recurring costs.

SaaS offer high scalability, which gives customers the option to accessmore, or fewer, services or features without a need to provision or buymore computers. When there is a need to update the software, rather thanpurchasing new version or update the version the customer own, customerscan rely on a SaaS provider to automatically perform updates and patchmanagement. This further reduces the burden on in-house IT staff. SinceSaaS applications are delivered over the Internet, users can access themfrom any Internet-enabled device and location. These advantages havemade SaaS a very popular alternative to packaged software that isinstalled on customer premises using customer hardware. The SaaSproviders may host the service on one or more servers in its privatedatacenter(s) or on one or more servers residing in one or more regionsin the public cloud.

Typically, the SaaS provider is identified by the domain name of itsservice (e.g. www.myworkday.com). Often, there is a different domainname associated with the servers that run public web page of the SaaSprovider (www.workday.corn) than the ones that runs the SaaS application(www.myworkday.com). This domain name can be resolved through a DNSquery to provide the IP address of the SaaS application server.

In case there are multiple servers, the DNS server may return adifferent IP address to two different requests that can be associatedwith different servers. The logic of the different name is from alocation basis. If the SaaS provider has several regions in the worldwhere it owns server, each requester will get back an IP server that iscloser to it. Inside the same region, the DNS service can still selectdifferent servers according to a load balancing point of view; the IPaddress that is returned is associated with a different server in thesame region. This latter case will return IP addresses for differentservers who usually share the same IP subnet.

The controller cluster of the virtual network in some embodiments keepsa table of known IP SaaS addresses. When the virtual network getspackets from a customer, the destination IP can be of three differenttypes. First, the packet can be associated with a private location ofthe entity (i.e., has a destination address in the private IP space ofthe entity). In this situation, the virtual network in some embodimentsroutes the packets to the corresponding compute node of the entity thatis associated with the packet's destination address.

Second, the packet has a destination addressed that is a public (notprivate) IP address that is not known to virtual network. These IPaddresses are referred to as generic public IP addresses. The virtualnetwork in some embodiments sends such a packet to the Internet from theingress virtual network node. Third, the packet has a destinationaddressed that is a public (not private) IP address known to virtualnetwork to be an IP address of a SaaS provider. Such IP addresses arereferred to as SaaS IP addresses. In some embodiments, such a packetwill be routed from a first virtual-network node (e.g., first CFE of afirst MFN) to a second virtual-network node (e.g., second CFE of asecond MFN) from where it is provided to the SaaS IP in the shortestpossible way in some embodiments.

FIG. 20 illustrates a process 2000 used by the central controllercluster of some embodiments to define routes for multi-homed SaaSproviders. This process identifies the various IP addresses associatedwith a SaaS service and identifies the shortest possible routes fromdifferent compute end nodes to one or more SaaS provider datacenters. Asshown, the process 2000 starts (at 2005) in some embodiments when thecontroller cluster receives a SaaS domain name list. In someembodiments, the SaaS domain list is provided by the administrator ofthe public cloud virtual network provider, while in other embodimentsthis list is provided by an administrator of the entity for which thepublic-cloud virtual network is defined by the virtual network provider.The table below provides an example of such a SaaS list.

Name 1. login.adaptiveinsights.com 2. adobeid-nal.services.adobe.corn 3.athenanet.athenahealth.com 4. login.bws.birst.com 5. account.box.com 6.centrify.com 7. identity.citrix.com 8. login.constantcontact.corn 9.account.docusign.com 10. login.github.com 11. secure.gooddata.corn 12.app.hubspot.com 13. login.huddle.net 14. hub.insidesales.com 15.login.marketo.com 16. system.netsuite.com 17. login.newrelic.com 18.login.microsoftonline.com 19. login.okta.com 20. login.oracle.com 21.myapps.paychex.com 22. login.salesforce.com 23.servicemax.cloudforce.com 24. hi.service-now.com 25.auth.tableausoftware.corn 26. login.ultimatesoftware.com 27.support.veeva.com 28. login.xero.com 29. www.zendesk.com

At 2010, the controller cluster stores the SaaS domain list in adatabase. In some embodiments, this database is accessible through oneor more interfaces (e.g., web server interface and/or API interface) toadministrators of the virtual network provider and/or of an entity(e.g., a tenant) for which the virtual network is deployed. Through thisinterface, an administrator can add or remove SaaS providers and/orassociated domain names to the list.

Next, at 2015, the controller cluster learns as many possible IPaddresses of SaaS servers associated with the domain on its list ofdomain names. To that end, the controller cluster in some embodimentsdirects different measurement agents 205 in the public clouds (that aredeployed by the VNP for one or more virtual networks deployed overdifferent public clouds) to execute DNS query for each domain name onthe list. Such a query is repeated periodically (e.g. every 30 minutes).The measurement agents 205 transfer back (at 2015) to the controllercluster the IP addresses that they learn for each domain name.

Different measurement agents may return different IP addresses as manySaaS are using geographical DNS service to match an adjacent server withthe client. The SaaS providers typically use Authoritative DNS Serversthat have lists of SaaS servers and their locations. When such a DNSserver gets a DNS request, it receives the measurement agent's IPaddress, and uses this IP address to Geo-IP map to identify the locationof the measurement agent and returns the IP address of the “best” serverfor the measurement agent. In some embodiments, the measurement agentalso provides the IP address of an end-compute node of the virtualnetwork, and the DNS server used by the SaaS provider provides an IPaddress based on the end-compute node's IP address.

The controller cluster stores (at 2020) in a database the returned IPaddresses along with their associated domain names. When at least somenumber of IPs (e.g., 5) belong to the same IP subnet (e.g., class Csubnet that includes 255 or less different addresses), the controllercluster adds the subnet itself to the database. In some embodiments,this database is accessible through one or more interfaces (e.g., webserver interface and/or API interface) to administrators of the virtualnetwork provider and/or of an entity (e.g., a tenant) for which thevirtual network is deployed. Through this interface, an administratorcan add or remove IP addresses. This interface also allows theaddition/removal of records associated with domain names that areadded/removed by an administrator. Also, in some embodiments, thecontroller cluster purges IP addresses that are not reported as beingused for a duration of time (e.g., every day, every several days, everyweek or every several weeks, etc.).

After 2020, the central controller identifies (at 2025) for eachreported IP address that is received from the reporting measurementagents in one or more public clouds (reporting regions), a set of publicclouds (nearby regions) that are near (i.e., within a threshold distanceof) the reporting region. In some embodiments, the nearness of tworegions are determined in terms of network distances that are measuredseparately between the regions. In some embodiments, the process 2000uses third party DNS services to identify an approximate location foreach IP address, and then uses the identified locations of the IPaddresses to quantify a distance between two IP addresses. The list ofthe set of regions identified for all the reported IP address isreferred to as IP vicinity report. When such operation is not done, theIP vicinity report will define all the virtual network regions as beingnear each IP address.

At 2030, the central controller provides the IP vicinity report to thedeployed measurement agents 205 that are deployed by the VNP for one ormore virtual networks deployed over different public clouds. Eachmeasurement agent then periodically measures (e.g., once every severalminutes or several hours) the distance between the measurement agent andeach SaaS provider IP address that is identified as being near themeasurement agent in the IP vicinity report. In some embodiments, themeasurement agent computes this distance to an IP address in terms ofthe delay for initiating a TCP connection with a server at this IPaddress. When the server, having this IP address is responding, the timeto that response is measured. Once a first response is accounted, themeasurement agent actively terminates the TCP connection in someembodiments. In some embodiments, the measurement agent also counts thenumber of successful TCP connection events and/or lost packets. Themeasurement agent in other embodiments uses other measurementtechniques, such as any one of the measurement techniques that weredescribed above.

At 2035, the controller cluster receives the distance measurements fromthe measurement agents. Next, at 2040, the controller cluster uses thereturned measurements (e.g., the delay and loss numbers reported fromeach measurement agent) to identify routes to each SaaS provider (e.g.,to each SaaS IP address) from each possible ingress MFN and/or from eachpossible MMCN. To identify the routes, the controller cluster performsshortest path route-identification process in some embodiments thatrelies on the weight values that are computed based on the measurementsto the different SaaS IP addresses, between different MFNs and to thedifferent MMCNs.

In some embodiments, the controller cluster repeats its routeidentification process periodically, or whenever the computed weightvalues change (e.g., based on new measurements or addition/deletion ofMFNs and SaaS IP addresses). In some embodiments, the controller clusterperforms the route identification operation (at 2040) for the multipleMMCNs and SaaS IP addresses together, as multiple egress nodesassociated with MMCNs and SaaS providers would be relevant inidentifying optimal routes to any one SaaS provider.

After identifying these routes, the controller cluster supplies (at2045) these routes to the MMCN edge nodes and the MFNs. For instance, insome embodiments, the controller cluster provides forwarding records(e.g., routing records that specify the next hop, routing records thatspecify the virtual network egress node, etc.) to the MMCN edge nodesand to the MFN CFEs. By using these forwarding records to perform theirrouting operations, the particular MMCN edge nodes and MFN CFEsimplement the optimal routes defined (at 2040) by the controllercluster. In some embodiments, the controller cluster supplies newrouting records to the MMCN edge nodes and the MFN CFEs whenever itidentifies new routes.

In some embodiments, the SaaS IP addresses that are discovered by theabove process are assumed to have a zero routing distance to the virtualnetwork node to which they connect (i.e., are assumed to be virtuallylocated in a public cloud region of the virtual network). In otherembodiments, the routing links between public cloud regions and SaaS IPaddresses have weights associated with them (as reflected in the exampleof FIG. 19), and these weights reflect the cost (e.g., measured delayand/or loss) associated with the path from those public cloud regions tothe SaaS IP addresses. Under this approach, the best regions to connectto a particular IP address are the regions from which the computedweight values (i.e., the cost measured in terms of packet delay andloss) are small.

One rational for associating a SaaS IP address with more than one MFNCFE in more than one public cloud region is that the distance of theSaaS server to multiple regions is much smaller than the typicaldistance between regions. In addition, it might cost less to routetraffic that is originating in one public cloud so it will stay till theegress node in the same cloud. In this case, the controller cluster insome embodiments binds each SaaS IP to at least one region in eachpublic cloud as long as the cost (e.g., the delay and loss) from thenearest region is below some cost (e.g., the delay and loss) threshold.When the route identification process needs to calculate a shortest pathto a certain IP address, it first looks to which regions this IP addressis bound, then it computes the shortest path from each egress node tothe bound region. In some embodiments, the routing tables themselves inthe routers do not need to include the external IP address as the datamessage will be carried in the tunnels until the destination egressnode, which then will look up to the IP address in the tunnel.

As mentioned above, the computed weight value in some embodimentsaccounts for the cost of packet delay and/or loss in a link between twopublic clouds regions, between a public cloud region and a SaaS providerdatacenter, or between a public cloud region and a tenant's compute endnode. In other embodiments, the computed weight value for each such linkis computed in terms of other types of parameters, such as the datathroughput costs that are charged by the public cloud providers and/orthe compute cost (for the compute elements used to implement the MFNcomponents in the public cloud) that are charged by the public cloudproviders.

In some embodiments, an entity can provide the VNP with input regardingwhich public clouds to use to deploy the entity's virtual network. Thisinput in some embodiments is also accompanied with a request for the VNPto create a dedicated virtual network for the entity (e.g., for theVNP's tenant) over one or more public clouds of one or more public cloudproviders. In some embodiments, the entity's input specifies the publiccloud providers to use and/or the public cloud regions in which thevirtual network should be defined. Conjunctively, or alternatively, thisinput in some embodiments specifies actual public cloud datacenters(PCDs) to use. The VNP in some embodiments supplements the set of publiccloud datacenters identified for an entity through the entity's inputwith one or more PCDs that the VNP identifies as desirable datacentersto add to the entity's set of datacenters.

FIG. 21 illustrates a process 2100 that the VNP uses in some embodimentsto deploy and configure dedicated MFNs to implement a dedicated virtualnetwork for an entity that requests such a network to be deployed over aparticular set of public cloud providers, a particular set of publiccloud regions, and/or a particular set of public cloud datacenters. Forthe entity, this process (1) identifies a set of public clouddatacenters of one or more public cloud providers to connect a set ofmachines of the entity, (2) deploys dedicated MFNs for the entity in theidentified set of public cloud datacenters, and then (3) configuresthese MFNs to implement the dedicated virtual network that connects theentity's set of machines across its identified set of public clouddatacenters.

This dedicated virtual network is defined over the compute and networkinfrastructure of one or more public cloud datacenters. In someembodiments, the dedicated MFNs that are specifically deployed for oneentity have the same attributes and perform the same operations (e.g.,check for a tenant identifier in performing its forwarding operations)as the shared MFNs that are deployed and used by multiple entities. Theonly difference between the dedicated and shared MFNs in theseembodiments is that the dedicated MFNs are used to process data messagesfor just one tenant, while the shared MFNs are used to process datamessages for multiple entities.

As mentioned above, an MFN in some embodiments is a conceptual groupingof several different components in a public cloud datacenter that withother MFNs (with other groups of components) in other public clouddatacenters establish one or more overlay virtual networks for one ormore entities. One of these components is a cloud forwarding element(CFE) that performs the next-hop forwarding needed to implement thepaths through the virtual network. Also, in some embodiments, each MFN'sgroup of components execute on different computers in the MFN's publiccloud datacenter, while in other embodiments, several or all of an MFN'scomponents execute on one computer of a public cloud datacenter. Thecomponents of an MFN in some embodiments execute on host computers thatalso execute other machines of other tenants, such as virtual machinesor containers of the VNP's tenants or other tenants of the public cloudproviders.

In some embodiments, the entity's machines that are connected by thevirtual network deployed by process 2100 are machines outside of anypublic cloud. In other embodiments, some of the entity's machines are ina public cloud, while other machines reside outside of the publicclouds. Also, in some embodiments, the entity's machines include SaaSprovider machines that the entity uses for certain SaaS operations.

Before receiving the entity's input, the process 2100 deploys (at 2105)measurement agents in one or more PCDs and/or PCD groups, and has theseagents exchange messages in order to generate network measurements thatquantify the quality of network connections between different pairs ofPCDs, different pairs of PCD groups (e.g., between different publiccloud regions or availability zones), and different pairs of PCD/PCDgroups. Examples of network measurements that the process generates fora connection between two PCDs, two PCD groups, and/or a PCD and a PCDgroup include loss, delay, and jitter experienced on this connection, aswell as the reliability and cost of the connection.

As described above, some embodiments measure the delay in a connectionlink between two PCDs and/or two PCD groups by measuring the round-triptime (RTT) for messages to be exchanged between two measurement agentsin the two PCDs and/or PCD groups. Jitter in some embodiments is thederivative of the delay (e.g., of RTT). Loss in some embodiments isderived from the number of packets dropped in a particular time frame.Reliability in some embodiments is derived from loss, e.g., a lossyconnection has a poor reliability score.

From the entity's network administrator, the process receives a request(at 2110) to deploy a dedicated virtual network. This request in someembodiments provides information regarding the location (e.g., networkaddresses) associated with the entity's machines that are locatedoutside and inside of the public clouds that need to be connectedthrough the virtual network. This location information includes theattributes (e.g., name, network address(es), etc.) of any office anddatacenter that include multiple machines.

The locations of individual end compute node machines in offices anddatacenters are not specifically identified in some embodiments, butrather are identified through their association with the offices anddatacenters. The provided location information in some embodiments canalso identify the location of one or more groups of one or more machinesin the public clouds, as well as the identities (and in some cases thelocations) of SaaS providers used by the entity.

In some embodiments, the request received at 2110 also specifies thepublic cloud providers to use and/or the public cloud regions in whichthe virtual network should be defined. Conjunctively, or alternatively,this input in some embodiments specifies actual public cloud datacentersto use. This input in some cases can specify multiple different publiccloud providers, and/or PCDs and/or PCD groups of different public cloudproviders.

Based on the input received at 2110, the process identifies (at 2115) aset of candidate PCDs and/or PCD groups over which the virtual networkcould be defined. In some embodiments, the identified set of PCDs and/orPCD groups include PCDs and/or PCD groups explicitly identified in theinput received at 2110, and/or are PCDs and PCD groups of public cloudproviders explicitly identified in this input. The process only includesthese explicitly identified PCDs and/or PCD groups in some embodiments.

However, in other embodiments, the process includes (at 2115) in theidentified set of candidate PCDs and/or PCD groups, other PCDs and/orPCD groups that might be desirable datacenters for the entity's virtualnetwork. Specifically, to provide recommendation to the entity, theprocess 2100 in some embodiments supplements the set of public clouddatacenters identified for an entity through the entity's input with oneor more other PCDs and/or PCD groups that the process identifies asdesirable PCD and/or PCD groups to add to the entity's set ofdatacenters.

To identify this other PCDs and/or PCD groups, the process in someembodiments uses pre-configured rules that identify desirable PCDsand/or PCD groups for certain locations and/or regions (e.g., for thelocations and/or regions specified by the entity and/or in which theentity's office and datacenters reside). These rules are defined in someembodiments based on previously deployed virtual networks for otherentities and/or based on measurements continuously taken by themeasurement agents that the VNP deploys in the different PCDs and/or PCDgroups.

From the current set of network measurement generated by the measurementagents for all the public clouds in which they are deployed, the processselects (at 2120) the subset of network measurements that are for thePCDs and/or PCD groups identified at 2115. In some embodiments, any onenetwork measurement (e.g., delay) for a connection link at any giventime is a weight-blended average of measurement values (e.g., delays)generated for that connection link over a past duration of time (e.g.,over the past few seconds, minutes, hours, days, etc.). Such an averagesmooths out the measurement values to ensure that the measurement valuesused are not too dependent on transient network conditions at any onegiven point in time.

Next, based on the identified PCDs and PCD groups, the process generates(at 2125) a routing graph by using one or more of the techniquesdescribed above by reference to FIGS. 3-5. At 2125, the process uses therouting graph to perform path searches that use the measurementsselected at 2120 to identify a set of paths connecting the entity'smachines (including SaaS machines to use) across the identified set ofPCDs. To account for mobile device connections, the path searches alsoidentify paths between all possible PCD pairs that can serve as ingressand egress nodes for paths across the dedicated virtual network.

To identify the paths, the process in some embodiments uses themeasurements selected at 2120 to perform shortest path searches. In someembodiments, the process 2100 performs two sets of paths searches. Afirst set of path searches only considers the PCDs and PCD groupsexplicitly identified in or through the entity's request received at2110. Another set of path searches in these embodiments not onlyconsiders the explicitly identified PCDs and PCD groups, but alsoincludes the PCDs and PCD groups that the process identifies aspotentially desirable at 2115.

As described above, the shortest path searches are based on weightsassigned to links of the routing graph, with the weights being derivedfrom the generated network measurements. Given that different networkmeasurements are used in different embodiments, the shortest pathsearches are smallest-cost searches that identify optimal paths betweenmachine endpoints (or between edge nodes) in the virtual network basedon any one or more of different criteria (such as delay, jitter, loss,reliability, etc.). As mentioned above, the cost search criteria in someembodiments also includes one or more financial costs (e.g., computecost, storage cost, networking cost) charged by the public cloudproviders.

Some embodiments allow different entities to direct the process to usedifferent combination of different types of criteria to cost the pathswhile it performs its path searches, e.g., one entity can direct theprocess to minimize message delay, another entity can direct the processto minimize message jitter, still another entity can direct the processto minimize loss and delay, etc. For each entity, the process in someembodiments custom configures its path search operations to optimize aset of criteria specified by the entity.

At 2130, the process determines whether the second set of path searches(that used both PCDs/PCD groups explicitly identified through the entityrequest received at 2110 and the other PCDs/PCD groups that the processidentified at 2115 as potentially desirable) resulted in any path thatwas better for connecting two of the entity's machine endpoints(including endpoints for connecting to mobile devices) than any pathproduced by the first set of path searches (that only used PCDs/PCDgroups explicitly identified through the entity request received at2110).

If not, the process deploys (at 2135) dedicated MFNs for the entity inthe PCDs and PCD groups that are used by the paths generated by thefirst set of path searches. These dedicated MFNs in some embodimentsonly forward data message flows for that entity. In some embodiments,the dedicated MFNs do not forward data message flows for any otherentity other than the entity for which the process 2100 is performed. Inother embodiments, these MFNs also carry VNP data message flows that theVNP needs to send and receive in order to manage the virtual network forthe particular entity. In some embodiments, the process deploys one MFNfor the entity in each PCD or PCD group used by the paths generated bythe first set of paths searches. In other embodiments, the process candeploy more than one MFN for the entity in a PCD or PCD group whenadditional MFNs are needed to handle the traffic load.

To deploy an MFN in a PCD/PCD group of a public cloud provider, theprocess 2100 in some embodiments uses the APIs of the PCD's public cloudprovider to direct the provider's server to add one or more machinesthat implement the different components of the MFN (e.g., its CFE,etc.). Similarly, other processes described herein deploy MFNs inPCDs/PCD groups by also using such APIs.

The process 2100 also configures (at 2135) these MFNs (e.g., theforwarding elements of these MFNs) to implement these paths by (1) usingthe identified paths to define forwarding rules (e.g., next hop records)that configure the MFNs and edge nodes (in multi-machine compute nodesof the entity) to forward data message flows along the different paths,and (2) distributing these forwarding rules to the MFNs and the edgenodes. To distribute the forwarding rules, the process in someembodiments provides the rules to a set of controllers, which thendistribute them to the MFNs and edge nodes. In some embodiments, theprocess configures the edge nodes by communicating with these edge nodes(e.g., through service APIs or some other mechanism), or by using BorderGateway Protocol (BGP) when the edge nodes are not directly accessibleby the VNP managers/controllers. After 2135, the process ends.

When the process 2100 determines (at 2130) that the second set of pathsearches resulted in at least one better path that uses one of the otherPCDs/PCD groups that the process identifies as being potentiallydesirable at 2115, the process generates (at 2140) a report for displaythrough a UI (user interface) management console or other interfaces(e.g., email, etc.), to provide a recommendation to the entityadministrator to add other PCDs and/or PCD groups to the list of publicclouds, PCDs or PCD groups explicitly identified by the entity in itsrequest. In some embodiments, the recommended PCDs/PCD groups can belongto different public cloud providers than those specified by the entity,and/or can be in different regions than the PCDs/PCD groups specified bythe entity.

The process 2100 in some embodiments also generates a report for displayin a user interface even before deploying the MFNs at 2135 when thesecond set of searches do not produce any better paths, so that theentity administrator can review information about the virtual networkthat is to be deployed and provide modifications. In other embodiments,the process 2100 does not generate any such report before deploying thevirtual network that it identifies based on the paths generated at 2125,but provides this report after the virtual network has been deployed.Also, in some embodiments, the process 2100 does not providerecommendations before deploying the dedicated virtual network, but onlyprovides recommendations after deploying the dedicated virtual networkto span the PCDs and/or PCD groups identified through the input receivedat 2110.

At 2145, the process determines whether the entity's administratoraccepted any recommendation that the report presented at 2140. If not,the process transitions to 2135 to deploy and configure dedicated MFNsfor implementing the paths generated through the first set of pathsearches. On the other hand, when the entity's administrator accepts oneor more of the recommendations to add one or more PCDs/PCD groups, theprocess deploys (at 2150) dedicated MFNs for the entity in the PCDs andPCD groups used by the paths generated by the second set of pathsearches. In some embodiments, the process deploys one MFN in each PCDor PCD group traversed by a path identified through the second set ofpath searches. In other embodiments, the process can deploy more thanone MFNs in such a PCD or PCD group when additional MFNs are needed tohandle the traffic load.

The process 2100 configures (at 2150) the deployed MFNs (e.g., the CFEsof the MFNs) to implement the paths generated by the second set of pathsearches. To configure these MFNs, the process (1) uses the identifiedsecond search-set paths to define forwarding rules (e.g., next hoprecords) that configure the MFNs and edge nodes (in multi-machinecompute nodes of the entity) to forward data message flows along thedifferent paths, and (2) distributes these forwarding rules to the MFNs(e.g., the CFEs of the MFNs) and the edge nodes (e.g., the forwardingelements of these MFNs and edge nodes).

To distribute the forwarding records, the process in some embodimentsprovides the forwarding records to a set of controllers, which thendistribute them to the MFNs and edge nodes. In some embodiments, theprocess configures the edge nodes by communicating with these edge nodes(e.g., through service APIs or some other mechanism), while in otherembodiments it uses BGP or other routing protocols when the edge nodesare not directly accessible by the VNP managers/controllers. After 2150,the process ends.

In some embodiments, the VNP processes use the same forwarding schemefor both dedicated and shared virtual networks that they deploy. Forinstances, in some embodiments, these processes use the same doubleencapsulating approach for both dedicated and shared virtual networks.Other embodiments, however, use different forwarding schemes fordedicated virtual networks. For instance, when deploying a dedicatedvirtual network that spans the PCDs/PCD groups of one public cloudprovider, some embodiments do not use the IP-based double encapsulationapproach described above, but rather route the data messages by using asingle encapsulation header as the tenant identifier is no longer usedfor the dedicated MFNs in some embodiments.

When performed for different entities, the process 2100 can produce verydifferent virtual networks for these entities as a starting input ofthis process is each entity's request to use a particular set of publiccloud providers, a particular set of PCDs or groups of PCDs. FIG. 22presents an example that illustrates three different virtual networks2202, 2204 and 2206 deployed over several public clouds in the UnitedStates for three different companies. In this example, the first twocompanies have offices only in San Francisco, Palo Alto and Los Angeles,but they end up with two very different virtual networks 2202 and 2204that span different datacenters as one picks Amazon's public cloudinfrastructure, while the other Google's public cloud infrastructure.The third company in this example has its dedicated virtual network 2206span San Francisco and Chicago Google PCDs and New York Amazon PCD.

Even after completing the process 2100 for an entity, one or more VNPprocesses monitor the network conditions and the virtual networkdeployed for the entity, and adjust or recommend adjustments to thevirtual network that may be needed. For instance, in some embodiments,these processes repeatedly check the network measurements that themeasurement agents iteratively generate. These network measurements areneeded as the quality of the connections between and to the publicclouds may change over time.

With new measurements, the VNP processes perform new path searches,which might result in new preferred paths through the public clouds,and/or deployment of new dedicated MFNs, for an entity. In someembodiments, the VNP processes make these changes automatically. Inother embodiments, these processes generate recommendations to use thesenew paths and/or deploy new MFNs, and provide these recommendations forpresentation through a UI management console or other interfaces (e.g.,email, etc.) to the network administrators of the entity.

The VNP processes also perform new path searches, adjust the virtualnetwork and/or generate new recommendations based on statistics thatthey collect from the dedicated MFNs that are deployed for an entity. Insome of these embodiments, the VNP processes perform the new pathsearches based on both the collected statistics from the MFNs and newmeasurements from the measurement agents. Based on these new pathsearches, the VNP processes in some embodiments provide recommendationto add one or more public clouds, PCDs and/or PCD groups. Based on theanalysis of the collected statistics and/or new measurements, the VNPprocesses in some embodiments also recommend removal of underutilizedpublic clouds, PCDs and/or PCD groups.

FIG. 23 conceptually illustrates a VNP process 2300 that producesrecommendations to add MFNs based on collected statistics and newmeasurements. In some embodiments, this process is performed iteratively(e.g., once every hour, few hours, a day, etc.) for each deployedvirtual network of each entity. As shown, the process 2300 initiallycollects (at 2305) statistics from the MFNs and edge nodes thatimplement the virtual network of the entity. The collected statisticsrelate to the forwarding of the data message flows (e.g., number of datamessages, number of bytes, delay in transmission of data messages, etc.)by the MFNs and edge nodes.

Next, at 2310, the process collects new measurement values generated bythe measurement agents deployed in the PCDs or PCD groups in which thevirtual network's MFNs are deployed (i.e., in the PCDs or PCD groupsover which the virtual network currently spans). These measurements arenetwork measurements for the connection between and to these PCDs and/orPCD groups. As mentioned above, any one network measurement (e.g.,delay) for a connection link at any given time in some embodiments is aweight-blended average of measurement values (e.g., delays) generatedfor that connection link over a past duration of time (e.g., over thepast few seconds, minutes, hours, days, etc.).

At 2315, the process identifies one or more candidate PCDs and/or PCDgroups to add to current PCDs and/or PCD groups used to implement thevirtual network, and collects network measurements that the measurementagents have generated for these new candidate PCDs and/or PCD groups.These measurements are network measurements for the connection betweenand to these PCDs and/or PCD groups.

In some embodiments, the candidate PCDs and/or PCD groups are PCDsand/or PCD groups that are nearby or in between the PCDs and/or PCDgroups that are currently used to implement the virtual network. Thecandidate PCDs and/or PCD groups in some embodiments are identified froma look-up table that associates each PCD or PCD group (e.g., PCD or PCDgroup currently used by the virtual network) with other candidate PCD orPCD groups, or each connected pair of PCD and/or PCD groups (where eachPCD/PCD group in a pair has an MFN exchange data message flows with anMFN in the other PCD/PCD group in the pair) with other candidate PCD orPCD groups. In these or other embodiments, the candidate PCDs and/or PCDgroups are identified by identifying all PCDs and/or PCD groups that arewithin a threshold distance of PCDs and/or PCD groups currently used bythe virtual network.

In some embodiments, the process 2300 does not identify new candidatePCDs/PCD groups for each current PCD/PCD group used by the virtualnetwork. Rather, in these embodiments, the process 2300 only identifiescandidate replacement PCDs/PCD groups for existing PCDs/PCD groups thatare over congested, performing poorly, or on paths that are performingpoorly. The process 2300 in some embodiments does not identify candidatePCDs or PCD groups that the entity has previously specifically rejectedfor the virtual network.

Based on the existing and candidate PCDs/PCD groups, the process creates(at 2320) a routing graph by using one of the approaches described aboveby reference to FIGS. 3-5. Next, at 2325, the process performs a new setof path searches to identify paths for connecting the entities machineendpoints (e.g., branch office, datacenters, mobile device accesslocations, etc.). To identify these paths, the process 2300 uses thepath search techniques described above.

At 2330, the process 2300 determines whether the new set of pathsearches resulted in any path that is better for connecting two of theentity's machine endpoints than any path currently used by the virtualnetwork. One path is better than another path in some embodiments if ithas a better cost (e.g., smaller delay, fewer dropped packets, lessjitter, more reliable, financial cost, etc.). The cost in someembodiments is a blended cost generated by blending multiple metricvalues (e.g., delay, dropped packets, jitter, reliability, financialcost, etc.). The metric values used in some embodiments are selected bythe VNP processes, while in other embodiments they are specified by theentity. Also, some embodiments use the statistics collected at 2305 asthe metric values or to generate the metric values for the currentpaths, while using the measurement generated by the measurement agentsto generate the metric values for one or more portion of the newlyidentified paths. Some embodiments also use the measurements of themeasurement agents to generate at least partially the metric values ofthe current paths.

When the process 2300 determines (at 2330) that each newly identifiedpath is worse than its corresponding current path for connecting thesame two entity machine endpoints, the process ends. Otherwise, theprocess generates (at 2335) a report for display through a UI (userinterface) management console or other interfaces (e.g., email, etc.),to provide a recommendation to the entity administrator to add one ormore candidate PCDs and/or PCD groups that improves the virtual networkperformance as gauged by the evaluated metric score.

The metric score in some embodiments includes the cost of the virtualnetwork deployment. In other embodiments, the metric score does notinclude the cost. However, even in some such embodiments, the process2300 provides (at 2335) a cost estimate for the virtual network with therecommendation that it provides to the network administrator of theentity. At 2335, the recommended PCDs/PCD groups can belong to differentpublic cloud providers than those specified by the entity, and/or can bein different regions than the PCDs/PCD groups specified by the entity.

At 2340, the process determines whether the entity's administratoraccepted any recommendation that the report presented at 2335. If not,the process ends. On the other hand, when the entity's administratoraccepts one or more of the recommendations to add one or more PCDs/PCDgroups, the process deploys (at 2345) new dedicated MFNs for the entityin the newly added PCDs and/or PCD groups. In some embodiments, theprocess deploys one MFN in each such PCD or PCD group. In otherembodiments, the process can deploy more MFNs in a PCD or PCD group whenadditional MFNs are needed to handle the traffic load.

The process 2300 configures (at 2345) the newly deployed MFNs (e.g., theCFEs of the newly deployed MFNs) to implement one or more of the newlyidentified paths that were better than one or more previously usedpaths. It also reconfigures (at 2345) previously deployed MFNs (e.g.,the CFEs of the previously deployed MFNs) to remove the older pathsbeing replaced. To configure the newly deployed MFNs and reconfigure thepreviously deployed MFNs, the process (1) defines forwarding rules(e.g., next hop records) that configure the MFNs and edge nodes (inmulti-machine compute nodes of the entity) to forward data message flowsalong the desired old and new paths, and (2) distributes theseforwarding rules to the old and new MFNs and the edge nodes.

To distribute the forwarding rules (e.g., next hop records), the processin some embodiments provides the forwarding rules to a set ofcontrollers, which then distribute them to the MFNs and edge nodes. Whenone or more old paths are removed, one or more previously deployed MFNsmay no longer be needed. In such a case, the process terminates theoperations of the unnecessary MFN(s). After 2345, the process ends.

FIG. 24 illustrates an example of adding a new PCD to a virtual network2400 to improve its performance. This figure shows two operationalstages 2405 and 2410 of the virtual network 2400. The first operationalstage 2405 shows that after its initial deployment, the virtual network2400 spans three PCDs of two public cloud providers in three cities, SanFrancisco, New York and Philadelphia. This virtual network connects twobranch offices 2422 and 2424 and one datacenter 2426 of a corporation.

The second operational stage 2410 shows the virtual network 2400 afterit has been modified to add a fourth PCD in Chicago. This PCD has beenadded to improve the performance of the path between the San Franciscobranch office 2422 and the New York branch office 2424. As shown in thefirst stage 2405, the point-to-point path between the San Francisco andNew York PCDs has a weight value (e.g., a delay value) of 12, while thepath from San Francisco to Chicago PCDs and the path from Chicago to NewYork PCDs has a combined weight value of 10. Hence, in this example, theChicago PCD is added after the virtual networks deployment to improvethe connection between the San Francisco and New York offices. However,in this example, the virtual network 2400 continues using the directconnection between New York and San Francisco offices for flows from NewYork to San Francisco.

As mentioned above, the VNP processes in some embodiments also recommendremoval of underutilized public clouds, PCDs and/or PCD groups based onthe analysis of the collected statistics and/or new measurements. FIG.25 conceptually illustrates a VNP process 2500 that producesrecommendations to remove one or more underutilized MFNs. In someembodiments, this process is performed periodically (e.g., once everyhour, few hours, a day, etc.) or iteratively for each deployed virtualnetwork of each entity. In some embodiments, the process 2500 is alsoperformed iteratively based on other events, e.g., significant changesin data traffic, operation outage of a PCD, etc.

As shown, the process 2500 initially collects (at 2505) statistics fromMFNs that implement the virtual network of the entity. The collectedstatistics relate to the forwarding of the data message flows (e.g.,number of data messages, number of bytes, delay in transmission of datamessages, etc.) by the MFNs during a particular time period. At 2510,the process 2500 analyzes the collected statistics to identify anyunderutilized MFNs of the virtual network. In some embodiments, anunderutilized MFN is an MFN that, in the particular time period,processes a number of data message flows that fall below a certainthreshold. In other embodiments, an underutilized MFN is an MFN that, inthe particular time period, forwards less than a threshold amount ofdata (e.g., threshold amount of bits per second).

At 2515, the process then determines whether it identified anyunderutilized MFN at 2510. If not, the process ends. Otherwise, theprocess examines (at 2520) removing the identified underutilized MFN(s)by performing path searches without the identified MFN(s). When morethan one underutilized MFN is identified, the process in someembodiments performs just one set of path searches by removing all theunderutilized MFNs and identifying a set of optimal paths without any ofthe removed MFNs. In such cases with multiple underutilized MFNs, theprocess 2500 in other embodiments performs different sets of pathsearches with different combinations of one or more utilized MFNsremoved. These different path searches enable the process to providedifferent recommendations based on different combinations ofunderutilized MFNs being removed.

Next, based on the path searches performed at 2520, the processgenerates (at 2525) a report for display through a UI (user interface)management console or other interfaces (e.g., email, etc.), to provideone or more recommendations to the entity administrator to remove one ormore existing PCDs and/or PCD groups that improved the virtual networkperformance (as gauged by the evaluated metric score) or improve thevirtual network cost.

In the embodiments that deploy one MFN in each PCD/PCD group, the PCD(s)and/or PCD group(s) identified in each recommendation are PCD(s) and/orPCD group(s) in which an underutilized MFN is deployed. In otherembodiments, each recommendation in some embodiments is provided byreference to removing a specific MFN (e.g., when more than one MFN canbe deployed in each PCD/PCD group). In some embodiments, the process2500 not only recommends removal of underutilized MFNs, but alsorecommends removal of other MFNs based on other criteria. For instance,in some embodiments, removal of an intermediate MFN might result inbetter speed or delay performance of the virtual network as theintermediate MFN slows down its performance.

In some embodiments, each provided recommendation (at 2525) isaccompanied with a cost estimate and/or one or more expected performancemetric for the virtual network if the recommendation is accepted. Forinstance, a recommendation could say that if the recommendation isaccepted and the virtual network is deployed based on thisrecommendation, the virtual network would cost $X less per month, butthe one or more paths between one or more pairs of machine endpointswould worsen by a certain percentage as gauged by one metric (e.g.,slower by as much as Y % more). When more than one path is affected bythe removal of a particular combination of one or more MFNs in aparticular recommendation, the recommendation also expresses how eachone of these paths is affected (e.g., by providing for each path ametric score percentage that expresses how much the path would be worseor better).

At 2530, the process determines whether the entity's administratoraccepted any recommendation that the report presented at 2525. If not,the process ends. On the other hand, when the entity's administratoraccepts one or more of the recommendations to remove one or morePCDs/PCD groups, the process removes (at 2535) any MFN that isdesignated for removal in the selected recommendation. When the removedMFN is implemented by one or more machines that are deployed in a PCD,the process 2500 removes the MFN by using the API of the PCD's publiccloud provider to direct the provider's server to remove the MFN.

At 2535, the process also reconfigures the remaining deployed MFNs(e.g., the CFEs of these MFNs) and/or edge nodes to implement one ormore new paths (identified at 2520) to connect one or more machineendpoints each of which was previously connected by a prior path thatused a removed MFN. Like the other above-described processes, theprocess 2500 in some embodiments reconfigures the MFNs and/or edge nodesby defining and distributing next hop forwarding rules (e.g., next hoprecords) to one or more remaining MFNs. After 2535, the process ends.

FIG. 26 illustrates an example of removing a PCD 2616 from a virtualnetwork 2600 in order to remove an underutilized MFN 2620 in this PCD.This figure shows two operational stages 2605 and 2610 of the virtualnetwork 2600. The first operational stage 2605 shows that after itsinitial deployment, the virtual network 2600 spans three PCDs 2612,2614, and 2616 of two public cloud providers in three cities, SanFrancisco, Los Angeles and San Diego. This virtual network connectsthree branch offices 2622, 2624, 2626 of a corporation.

The second operational stage 2610 shows the virtual network 2600 afterit has been modified to remove the PCD 2616 in San Diego. This PCD hasbeen removed as its MFN 2620 is underutilized. With this PCD removed,the San Diego office 2626 connects to the PCD 2614 in Los Angeles. Asshown in the second stage 2610, the path between the Los Angeles and SanDiego offices 2624 and 2626 is 10% slower, but the monthly cost of thevirtual network has been reduced by 30%. This is an acceptable tradeoffgiven the small number of data message flows to and from the San Diegooffice.

The VNP processes of some embodiments allows an entity with a dedicatedvirtual network to use other MFNs that have not been specificallydeployed for the entity under certain circumstances. These other MFNs insome embodiments are shared MFNs as they are used to deploy differentvirtual networks for different entities, as opposed to the dedicatedMFNs that are specifically deployed for the single entity to deploy justthe dedicated virtual network for this entity. One example of usingshared MFNs for an entity with a dedicated virtual network is foralleviating congestion at a particular location in the dedicated virtualnetwork.

Specifically, the VNP processes of some embodiments allow for temporaryusage of one or more shared MFNs when a portion of the entity'sdedicated virtual network (e.g., one or more dedicated MFNs of theentity) appears congested or is expected to be congested. In someembodiments, these shared MFNs are part of a shared virtual network thatthe VNP provider deploys to handle traffic overflow for all dedicatedvirtual networks that it has deployed. In other embodiments, the sharedMFNs not only handles the overflow data traffic of one or more entitieswith dedicated virtual networks, but are also used to defined multipleseparate virtual networks for multiple other entities, with each ofthese virtual networks passing the primary data message traffic of itsassociated entities. Examples of such shared virtual networks weredescribed above by reference to FIGS. 1-20.

FIG. 27 conceptually illustrates a VNP process 2700 that producesrecommendations to offload some of the data message traffic from adedicated virtual network of the entity to one or more shared MFNs forat least a portion of at least one path between at least one pair ofmachine endpoints. In some embodiments, this process is performediteratively (e.g., once every hour, few hours, a day, etc.) for eachdeployed virtual network of each entity.

As shown, the process 2700 initially collects (at 2705) statistics fromMFNs that implement the virtual network of the entity. The collectedstatistics relate to the forwarding of the data message flows (e.g.,number of data messages, number of bytes, delay in transmission of datamessages, etc.) by the MFNs during a particular time period. At 2710,the process 2700 analyzes the collected statistics to identify one ormore overutilized MFNs of the virtual network. In some embodiments, anoverutilized MFN is an MFN that, in the particular time period, forwardsa number of data messages or an amount of data messages that is greaterthan a certain threshold.

At 2715, the process then determines whether it identified anyoverutilized MFN at 2710. If not, the process ends. Otherwise, theprocess performs (at 2720) path searches to examine adding one or morepaths that divert data message flows from each identified overutilizeddedicated MFN to one or more shared MFNs (i.e., MFNs used to deploymultiple virtual networks for multiple entities), in order to reduce theload on the overutilized MFN. To divert only some of the data messageflows away from the overutilized MFN, the path search in someembodiments defines weights (i.e., thresholds, etc.) to control theamount of data message flows that should be redirected from the existingoverutilized MFN to a candidate shared MFN.

To identify each diverting path, the method identifies one or morecandidate shared MFNs to divert some portion of the traffic for eachidentified, overutilized dedicated MFNs of the entity. In someembodiments, a candidate, shared MFN is in the same PCD or PCD group asthe overutilized, dedicated MFN for which it is identified when such acandidate, shared MFN is available. When no such shared MFN candidate isavailable, the VNP process 2700 deploys one such MFN in the PCD or PCDgroup of the overutilized, dedicated MFN, or selects an existing sharedMFN (or deploys on such MFN) in a PCD or PCD group that is nearby thePCD or PCD group of the overutilized, dedicated MFN.

Next, based on the path searches performed at 2720, the processgenerates (at 2725) a report for display through a UI (user interface)management console or other interfaces (e.g., email, etc.), to provideone or more recommendations to the entity administrator to redirect someof the data message flows away from one or more overutilized dedicatedMFNs to the shared virtual network. In some embodiments, each providedrecommendation (at 2725) is accompanied with a cost estimate and/or aperformance expectation for the virtual network if the recommendation isaccepted. For instance, a recommendation could say that if therecommendation is accepted, the virtual network would cost $X more orless per month, and improve the connection (as expressed by one or moreof several metrics) between at least one particular pair of endpointmachines by Y %. When the adjustment would improve the performance oftwo or more connection between two or more endpoint-machine pairs, therecommendations in some embodiments express the improvement for eachpair separately.

At 2730, the process determines whether the entity's administratoraccepted any recommendation that the report presented at 2725. If not,the process ends. On the other hand, when the entity's administratoraccepts one or more of the recommendations to divert some of the datamessage flows to the shared virtual network, the process defines (at2735) new configuration data (1) to reconfigure the dedicated virtualnetwork to divert some of the data message flows to the shared MFNs, and(2) to configure the shared MFNs (e.g., the CFEs of the shared MFNs) toforward the redirected data message flows to their destinations or backto the dedicated virtual network at another edge node of the dedicatedvirtual network.

In some embodiments, the process 2700 reconfigures the dedicated virtualnetwork by configuring one or more front-end load balancers to divertsome of the data message flows to shared MFNs that are used to deploymultiple shared virtual networks for multiple entities. This process insome embodiments configures the shared MFNs (e.g., the CFEs of theshared MFNs) by using the newly defined paths to define forwarding rules(e.g., next hop forwarding rules), which it then distributes (throughcontrollers) to the shared MFNs to configure these MFNs.

In some embodiments, the process provides such forwarding rules to thefront-end load balancers to configure their operations, along withweight values that specify how much of the data message flows the loadbalancers should divert to the shared MFNs. In other embodiments, theprocess 2700 reconfigures the dedicated virtual network by configuringone or more dedicated MFNs (e.g., the CFEs of the MFNs) with new WCMP(weight cost multipathing operation) parameters that control how theseMFNs distribute the flows to the overutilized dedicated MFN and toalternative shared MFN(s), in order to reduce the load of theoverutilized dedicated MFN. After 2735, the process ends.

FIG. 28 illustrates an example of directing some of the data messageflows away from an over congested dedicated MFN 2820 in a dedicatedvirtual network 2800 to a shared virtual network 2850. This figure showstwo operational stages 2805 and 2810 of the virtual network 2800 thatconnects three offices 2880, 2882 and 2884 of an entity in Los Angeles,Chicago and New York. The first operational stage 2805 shows that thatthe dedicated MFN 2820 in a Chicago PCD 2814 as being over congested.This MFN connects dedicated MFNs 2822 and 2824 in a Los Angeles PCD 2812and a New York PCD 2816.

The second stage 2810 shows that the dedicated MFN 2822 reconfigured inthe Los Angeles PCD 2812 to redirect the data message flows from the LosAngeles dedicated MFN 2822 to the New York dedicated MFN 2824 through ashared MFN 2830 in the same Chicago PCD 2814 as the overutilizeddedicated MFN 2820. The shared MFN 2830 is part of the shared virtualnetwork 2850, which is used by a second company to connect its Chicagoand New York offices 2870 and 2872.

In this example, this shared MFN 2830 in Chicago forwards these datamessages to the MFN 2832 in New York, which then provides them to thededicated MFN 2824 in New York. In other examples, the shared MFN 2830in Chicago can forward these data messages to the dedicated MFN 2824 inNew York. Also, in the example in FIG. 28, the Chicago PCD 2814 is stillused to exchange other messages between the LA and Chicago offices 2880and 2882, between the NY and Chicago offices 2884 and 2882, and from NewYork office 2884 to the LA office 2880.

In some embodiments, the data message flows from LA to New York areforwarded to the shared MFN 2830 in Chicago by a load balancer, whichcan be separate from the LA MFN 2822, or part of this MFN 2822. In stillother embodiments, this load balancing operation is part of WCMPoperation that the cloud forwarding element of the MFN 2822 performs.Yet other embodiments deploy this load balancer in the Chicago PCD 2814to distribute some of the load between the dedicated MFN 2820 and theshared MFN 2830. Instead of just forwarding all the flows from LA to NewYork through the shared virtual network, other embodiments use othertechniques to distribute the load between the dedicated MFN 2820 andshared MFN 2830 in Chicago. For instance, the MFNs in LA and New Yorkperform WCMP to distribute the flows that they send to Chicago amongthese two MFNs, i.e., use weights to specify percentage of flows thatthey direct to each of these MFNs.

In FIG. 28, the shared MFN 2830 is an intermediary node that redirectssome of the flows away from the dedicated MFN 2820 but delivers theseflows back to the dedicated virtual network at another dedicated MFN2824. In other examples, the shared virtual network delivers theredirected data message flow to an entity's machine endpoint, e.g., anentity's office or branch location. FIG. 29 illustrates one suchexample. This example is similar to the example of FIG. 28, except thatthe shared MFN 2832 in New York provides the flows from the LA officedirectly to the New York office 2884 without first going through thededicated MFN 2824 in New York.

In yet other examples, one or more shared MFNs of one or more sharedvirtual network are used in some embodiments as a way to relieve flowcongestion at multiple different locations along a path through adedicated virtual network. FIG. 30 illustrates one such example. Thefirst stage 3005 in this figure shows a path making five hops along adedicated virtual network 3000 that uses five PCDs 3011, 3012, 3014,3016, and 3018 in Tokyo, Los Angeles, New York, Barcelona and Tel Avivto connect two offices 3080 and 3082 in Tokyo and Tel Aviv. It alsoshows the MFNs 3020 and 3024 in LA and Barcelona as being used toprocess too many flows.

The second stage 3010 shows some of the data message flows exchangedbetween the Tokyo and Tel Aviv offices are redirected in the LA PCD 3012and Barcelona PCD 3016 to shared virtual network 3090. In redirectingthese flows, the Tokyo MFN 3021 is configured to forward certain numberof flows to the shared MFN 3060 in the LA PCD 3012. The New York sharedMFN 3062 receives these redirected flows and directs them back to thededicated virtual network 3000 at the dedicated MFN 3022 in the New YorkPCD 3014.

As the dedicated virtual network 3000 is congested in Barcelona alongthe path from Tokyo to Tel Aviv, a dedicated MFN 3024 in Barcelona againredirects some of the data message flow to the shared virtual network3090 by directing these flows to a shared MFN 3064 in the Barcelona PCD3016. The Barcelona shared MFN 3064 receives these redirected flows,forwards them to a Tel Aviv shared MFN 3066, which then directs themback to the dedicated virtual network 3000 by forwarding them to the TelAviv dedicated MFN 3026.

In the example of FIG. 30, the redirection is accomplished in someembodiments by using load balancers, while in other embodiments it isaccomplished through a set of weights that the cloud forwarding elementsof the MFNs use to perform WCMP operations for data message flows fromthe Tokyo office 3080 to the Tel Aviv office 3082. Also, in thisexample, the offloading shared MFNs of the shared virtual network are inthe same datacenters as their corresponding dedicated MFNs. As mentionedabove, this does not have to be the case (i.e., the data message flowscan be redirected to other MFNs in other datacenters, such as nearbydatacenters).

After configuring one or more shared MFNs to handle some of the datamessage traffic for an entity's dedicated virtual network, the VNPprocesses of some embodiments continue monitoring the MFNs of thededicated virtual network to determine whether the load on previouslycongested MFNs has fallen sufficiently such that one or more of theshared MFNs are no longer needed. FIG. 31 illustrates one such process3100. This process is iteratively performed by the VNP machines afterthe process 2700 detects that at least one dedicated MFN for the entityis over congested and some or all of the traffic should be redirectedaway from it to one or more shared MFNs.

As shown, the process 3100 initially collects (at 3105) statistics fromMFNs that implement the dedicated virtual network of the entity. Thecollected statistics relate to the forwarding of the data message flows(e.g., number of data messages, number of bytes, delay in transmissionof data messages, etc.) by the MFNs during a particular time period. At3110, the process 3100 analyzes the collected statistics to determinewhether it should reclassify as not congested any dedicated MFNs thatthe process 2700 previously identified as over congested. For thisreclassification of a previously identified overutilized MFN, theprocess 3100 in some embodiments requires the data message load on theMFN (e.g., the number of data message flows processed, or the overallnumber of bytes processed, by the MFN) to be below a certain thresholdfor a duration of time (e.g., for multiple iterations of the process3100). This is done in order to ensure that one MFN does not constantlyswitch between an over congested status and a uncongested status.

At 3115, the process then determines whether it changed (at 3110) thestatus of any MFN from congested to not congested. If not, the processends. Otherwise, the process reconfigures (at 3120) the dedicatedvirtual network to no longer redirect any data message traffic away fromany dedicated MFNs that had its status changed (at 3110) from congestedto not congested. In the embodiments in which the process 2700 usedfrontend load balancers to redirect some of the data message flow awayfrom congested dedicated MFNs, the process 3100 does its reconfiguringby distributing new load balancing criteria to any frontend loadbalancer that the process 2700 deployed for any reclassified MFN (at3110).

On the other hand, in the embodiments in which the process 2700redirects part of the data message flow away from a congested dedicatedMFNs by distributing WCMP parameters to the forwarding elements of theMFNs and edge nodes, the process 3100 does its reconfiguring (at 3120)by distributing new WCMP parameters to these cloud forwarding elementsand edge nodes that forward data messages to a reclassified MFN (at3110). These new WCMP parameters cause these forwarding elements to notredirect any data message flows away from the reclassified MFN. At 3120,the process 3100 also removes forwarding records from any shared MFN(e.g., the CFE of any shared MFN) of the VNP shared virtual network thatare no longer needed to forward redirected traffic of the dedicatedvirtual network. After 3120, the process ends.

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral or transitory signals.

FIG. 32 conceptually illustrates a computer system 3200 with which someembodiments of the invention are implemented. The computer system 3200can be used to implement any of the above-described hosts, controllers,and managers. As such, it can be used to execute any of the abovedescribed processes. This computer system includes various types ofnon-transitory machine readable media and interfaces for various othertypes of machine readable media. Computer system 3200 includes a bus3205, processing unit(s) 3210, a system memory 3225, a read-only memory3230, a permanent storage device 3235, input devices 3240, and outputdevices 3245.

The bus 3205 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of thecomputer system 3200. For instance, the bus 3205 communicativelyconnects the processing unit(s) 3210 with the read-only memory 3230, thesystem memory 3225, and the permanent storage device 3235.

From these various memory units, the processing unit(s) 3210 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments. Theread-only-memory (ROM) 3230 stores static data and instructions that areneeded by the processing unit(s) 3210 and other modules of the computersystem. The permanent storage device 3235, on the other hand, is aread-and-write memory device. This device is a non-volatile memory unitthat stores instructions and data even when the computer system 3200 isoff. Some embodiments of the invention use a mass-storage device (suchas a magnetic or optical disk and its corresponding disk drive) as thepermanent storage device 3235.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 3235, the system memory 3225 is a read-and-write memorydevice. However, unlike storage device 3235, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 3225, the permanent storage device 3235, and/or theread-only memory 3230. From these various memory units, the processingunit(s) 3210 retrieve instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 3205 also connects to the input and output devices 3240 and3245. The input devices enable the user to communicate information andselect commands to the computer system. The input devices 3240 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 3245 display images generated by thecomputer system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 32, bus 3205 also couples computer system 3200to a network 3265 through a network adapter (not shown). In this manner,the computer can be a part of a network of computers (such as a localarea network (“LAN”), a wide area network (“WAN”), or an Intranet, or anetwork of networks, such as the Internet. Any or all components ofcomputer system 3200 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra-density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For instance, several of theabove-described examples illustrate virtual corporate WANs of corporatetenants of a virtual network provider. One of ordinary skill willrealize that in some embodiments, the virtual network provider deploysvirtual networks over several public cloud datacenters of one or morepublic cloud providers for non-corporate tenants (e.g., for schools,colleges, universities, non-profit entities, etc.). These virtualnetworks are virtual WANs that connect multiple compute endpoints (e.g.,offices, datacenters, computers and devices of remote users, etc.) ofthe non-corporate entities.

Several embodiments described above include various pieces of data inthe overlay encapsulation headers. One of ordinary skill will realizethat other embodiments might not use the encapsulation headers to relayall of this data. For instance, instead of including the tenantidentifier in the overlay encapsulation header, other embodiments derivethe tenant identifier from the addresses of the CFEs that forward thedata messages, e.g., in some embodiments in which different tenants havetheir own MFNs deployed in the public clouds, the tenant identity isassociated with the MFN's that process the tenant messages.

Also, several figures conceptually illustrate processes of someembodiments of the invention. In other embodiments, the specificoperations of these processes may not be performed in the exact ordershown and described in these figures. The specific operations may not beperformed in one continuous series of operations, and different specificoperations may be performed in different embodiments. For instance, foran entity, some embodiments deploy some or all of the MFNs in PCDs/PCDgroups before paths are generated for the entity, even though some ofthe above-described figures illustrate the deployment of the MFNs afterthe path searches. Furthermore, the process could be implemented usingseveral sub-processes, or as part of a larger macro process.

Several embodiments described above allow a dedicated virtual networkthat is deployed for one entity to use one or more shared MFNs when aportion of the entity's dedicated virtual network (e.g., one or morededicated MFNs of the entity) appears congested or is expected to becongested. Other embodiments deal with such congestions differently. Forinstance, some embodiments deploy additional dedicated MFNs (i.e.,redundant MFNs) to portions of the dedicated virtual network that arecongested or expected to be congested. Some of these embodiments thenconfigure forwarding elements (e.g., frontend load balancers) in thePCDs, edge nodes at MMCNs, and other dedicated MFNs to distribute loadamong a cluster of duplicate (e.g., redundant) MFNs. Thus, one ofordinary skill in the art would understand that the invention is not tobe limited by the foregoing illustrative details, but rather is to bedefined by the appended claims

The invention claimed is:
 1. A method of operating a virtual network fora particular entity over a set of two or more public cloud datacenters,the method comprising: defining a larger, first virtual network over alarger, first set of the public cloud datacenters; defining, for theparticular entity, a smaller, second virtual network over a smaller,second set of the public cloud datacenters that does not include all thepublic cloud datacenters in the first set; using the second virtualnetwork to forward a first set of data messages associated with externalmachines of the particular entity outside of any public clouddatacenters; and based on a detected condition, using the first virtualnetwork to forward a second set of data messages associated with theexternal machines of the particular entity through at least part of thetraversal of the second set of data messages through the public clouddatacenters.
 2. The method of claim 1, wherein the first virtual networkis a shared virtual network used by multiple entities, while the secondvirtual network is a dedicated virtual network used by the particularentity.
 3. The method of claim 1, wherein defining the larger, firstvirtual network comprises deploying and configuring a first set offorwarding elements in the larger, first set of the public clouddatacenters; defining the smaller, second virtual network comprisesdeploying and configuring a second set of forwarding elements in thesmaller, second set of the public cloud datacenters; using the secondvirtual network comprises using forwarding elements in the second set toforward the first set of data messages; using the first virtual networkcomprises using forwarding elements in the first set to forward thesecond set of data messages.
 4. The method of claim 3, wherein theforwarding elements in the first set of forwarding elements are sharedforwarding elements that are used to forward data message flows formultiple entities, while the forwarding elements in the second set offorwarding elements are dedicated forwarding elements that are used toforward data message flows for the particular entity.
 5. The method ofclaim 1, wherein a first set of forwarding elements in the first set ofpublic cloud datacenters implement the first virtual network and asecond set of forwarding elements in the second set of public clouddatacenters implement the second virtual network, and the detectedcondition is congestion at a particular forwarding element used toimplement the second virtual network.
 6. The method of claim 5 furthercomprising collecting statistics regarding data message flows processedby the second set of forwarding elements; based on the collectedstatistics, determining that a particular forwarding element in thesecond set of forwarding elements is congested; wherein using the firstvirtual network comprises redirecting a set of the data message flowsaway from the particular forwarding element to a forwarding element in afirst set of forwarding elements.
 7. The method of claim 6, whereinredirecting the set of the data message flows comprises configuring aload balancer to redirect the set of the data message flows away fromthe particular forwarding element to the forwarding element in a firstset of forwarding elements.
 8. The method of claim 7, whereinconfiguring the load balancer comprises configuring the load balancer toredirect 1 out of N new data message flows away from the particularforwarding element, wherein N is an integer.
 9. The method of claim 6,wherein collecting statistics comprises: generating, at the forwardingelements, statistics regarding data message flows that the forwardingelements process; collecting, at a controller, the generated statisticsfrom the forwarding elements.
 10. The method of claim 1 furthercomprising before defining the second virtual network, receiving anidentification of the second set of the public cloud datacenters fromthe entity as the set of public cloud datacenters over which the secondvirtual network should be defined.
 11. The method of claim 1, whereinthe first virtual network is administered by a virtual network provider(VNP) that deploys multiple different virtual networks for multipledifferent entities of a plurality of public cloud datacenters.
 12. Themethod of claim 11, wherein the first virtual network is for the VNP tomanage all a set of virtual networks deployed for a set of entities bythe VNP.
 13. The method of claim 1 further comprising defining first andsecond sets of paths through the first and second virtual networks basedon first and second sets of path-defining criteria, wherein the firstset of path defining criteria for the first virtual network is differentthan the second set of path defining criteria for the second virtualnetwork.
 14. The method of claim 1 further comprising deploying andconfiguring forwarding elements in a plurality of public clouddatacenters in order to define the first and second virtual networks,wherein at least one forwarding element for the first virtual network isdeployed and configured in a public cloud in which a forwarding elementfor the second virtual network cannot be deployed.
 15. The method ofclaim 1 further comprising: collecting statistics regarding the secondset of data messages that are forwarded through the first virtualnetwork; generating a bill, based on the collected statistics, for theparticular entity to pay in order to account for the particular entity'susage of the first virtual network.
 16. A system for operating a virtualnetwork for a particular entity over a set of two or more public clouddatacenters, the system comprising: a first set of managed forwardingelements deployed in a first set of public cloud datacenters (PCDs) todefine a first virtual network across the first PCD set; a second set ofmanaged forwarding elements deployed in a second set of PCDs to define asecond virtual network across the second PCD set, the second PCD set notincluding all PCDs in the first PCD set, the second virtual network usedto forward a first set of data messages associated with externalmachines of the particular entity outside of the public clouddatacenters; and a set of servers to detect a condition and in responseto use the first virtual network to forward a second set of datamessages associated with at least one external machine of the particularentity for at least part of the second set of data messages' traversalthrough the PCDs.
 17. The system of claim 16, wherein the first virtualnetwork is a shared virtual network used by multiple entities, while thesecond virtual network is a dedicated virtual network used by theparticular entity.
 18. The system of claim 16, wherein the forwardingelements in the first set of forwarding elements are shared forwardingelements that are used to forward data message flows for multipleentities, while the forwarding elements in the second set of forwardingelements are dedicated forwarding elements that are used to forward datamessage flows for the particular entity.
 19. The system of claim 16,wherein the detected condition is congestion at a particular forwardingelement used to implement the second virtual network.
 20. The system ofclaim 16, wherein the second set of the PCDs are identified by theentity as the set of PCDs over which the second virtual network shouldbe defined.